{"id":"dcf58bdf-3e15-4635-b075-2ff73106eb9d","arxiv_id":"2411.14100","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BEST-STD trains a bidirectional Mamba encoder with contrastive learning and vector quantization to make speaker-agnostic speech tokens for fast spoken term detection.","lead":"The authors propose BEST-STD, a speech tokenizer that converts audio into discrete, speaker-agnostic tokens, enabling spoken term detection with fast text-based search. They report better accuracy than several baselines on LibriSpeech and TIMIT, though the efficiency advantage is asserted rather than measured.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central accuracy claim rests on an under-specified retrieval protocol: queries are extracted from the archive itself and the paper never states that a query's source segment is excluded, so reported MAP/MRR may be inflated by trivial self-matches. Runtime claims are also unmeasured.","rationale":"The reader identified DTW alignment as the weakest assumption, but the empirical evidence in Table I mitigates that concern: the method achieves high same-word Jaccard similarity, and Table II shows strong discrimination, suggesting the training signal is adequate. The more consequential vulnerability is the evaluation protocol. Because queries are harvested from the same archive used for retrieval, and the paper never says the source segment is excluded, the near-perfect agreement between a query and its own source segment could dominate the rankings. This would inflate all methods, but the absolute magnitudes reported (MAP 0.86) are only meaningful if self-matches are handled. The paper also omits the hop length h, so segment-level ground truth cannot be reconstructed. Finally, the abstract promises 'more efficient' retrieval, but no timing experiments appear; the only support is a qualitative comparison to DTW. Since the central claim is twofold (accuracy and efficiency), both pillars need support. The verdict should remain CONDITIONAL, but the conditions should require a re-run with source-segment exclusion and timing measurements, rather than (or in addition to) additional baselines.","tokens_in":9115,"tokens_out":9207,"duration_ms":87501,"concrete_test":"Re-run the LibriSpeech retrieval evaluation after excluding, for each of the 600 queries, every archive segment that overlaps the query's source utterance (or at least the exact source segment), and recompute MAP/MRR/MTWV from Section IV.B. Also report the segment hop h and the average number of relevant segments per query. If the metrics drop materially relative to Table II, the reported superiority is an artifact of self-retrieval rather than tokenization quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is the retrieval evaluation protocol. Section III.A states 'For testing, we extracted spoken terms from the train-clean-100 set', which is the same set used as the archive. The retrieval algorithm (Section II.D) scores candidate segments against the query token sequence using Jaccard similarity over bigrams. The paper never states that the query's source segment or source utterance is removed from the candidate list or from ground truth. Since the query is literally a segment (or is contained in a segment) of an archive track, that source segment will contain the exact query tokens and will be retrieved with near-perfect similarity. Thus every query has a guaranteed self-match, potentially inflating MAP/MRR for all systems, and making the reported BEST-STD scores (e.g., MAP 0.86/0.83 on LibriSpeech) not interpretable. The same section also fails to specify the segment hop h and the definition of segment-level relevance, so the ground truth is ambiguous. Additionally, the headline efficiency claim ('significantly faster retrieval times') is supported by no runtime measurement anywhere in the paper. A fair re-evaluation that excludes self-matches and reports h and timing would determine whether the accuracy and efficiency claims hold.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BEST-STD, a query-by-example spoken term detection system built on discrete speech tokens. Speech is mapped to frame embeddings by a bidirectional Mamba encoder, trained with a self-supervised contrastive objective in which DTW-aligned frames from different utterances of the same word form positive pairs, plus a vector-quantization commitment loss. At test time, archive audio is segmented and tokenized into bigram sequences indexed in an inverted index; a query is tokenized and scored against candidate segments by maximum Jaccard similarity. Experiments on LibriSpeech and TIMIT compare the method against DTW-based feature baselines and existing speech tokenizers, reporting higher Jaccard consistency of token sequences and higher MAP/MRR/MTWV, as well as a claim of faster retrieval.","tokens_in":9391,"tokens_out":5443,"duration_ms":54922,"significance":"If the empirical claims hold, the paper would make a useful contribution by showing that discrete, speaker-agnostic speech tokens can support effective query-by-example search with an inverted-index retrieval scheme, avoiding expensive DTW at search time. The self-supervised objective is clearly specified, the comparison covers several representative baselines, and the authors provide a public code URL, which are strengths. The token-consistency analysis, however, largely measures the property that the training objective directly optimizes, so it is not an independent validation of speaker invariance; the independent evidence is the retrieval comparison in Table II, whose current evaluation protocol is under-specified in a way that affects the interpretability of the headline numbers.","major_comments":[{"comment":"The evaluation protocol for Table II is under-specified in a way that directly affects the headline accuracy claim. Section III.A states that spoken terms for testing were extracted from the train-clean-100 set, which is the same set used as the speech archive. Section II.D scores every candidate segment against the query token sequence, and the paper never states that the query's source segment or source utterance is removed from the candidate list or from the ground truth. Because each query is a segment of an archive track, the source segment will contain the exact query tokens and will be retrieved with near-perfect Jaccard similarity, giving every query a guaranteed self-match. This inflates MAP/MRR for all systems and makes the numbers in Table II (e.g., MAP 0.86/0.83) not interpretable as detection performance; please state whether self-matches were excluded and, if not, re-run the evaluation with the source segment/utterance removed and report the corrected numbers.","section":"III.A and II.D"},{"comment":"The retrieval setup is not reproducible because the segment hop h and the segment-level relevance criterion are never specified. Section II.C defines overlapping segments by length l and hop h, but no value for h is given; Section III.D states that 1-second (l) segments were used for training, but does not state the archive indexing parameters. It is also unclear whether a candidate segment is relevant if it overlaps the query, contains the full word, or satisfies some other condition, and how multiple overlapping relevant segments contribute to MAP/MRR/MTWV. Please specify h, the relevance definition, and the pooling rule, and report results under that protocol.","section":"II.C and III.D"},{"comment":"The efficiency claim is unsupported. The text states that the inverted index delivers \"significantly faster retrieval times compared to DTW-based methods,\" but no runtime, memory, or complexity measurements are reported anywhere. Please provide retrieval-time measurements on the same archive, or a clear complexity analysis, to support the efficiency claim.","section":"IV.B"},{"comment":"The training objective assumes that the DTW alignments in Eq. (4) are semantically correct. If the alignments are noisy, the anchor-positive pairs selected in Eq. (5) may pair frames that are acoustically similar but phonetically unrelated, and the model could learn spurious cross-speaker correspondences. The paper provides no validation of DTW alignment quality or an ablation against alternative alignment strategies; this is a load-bearing assumption for the claimed speaker-agnostic tokenization and should be addressed, for example by measuring alignment consistency or comparing with a simpler fixed-alignment baseline.","section":"II.B"}],"minor_comments":[{"comment":"There are typos and capitalization inconsistencies, including \"retreiving,\" \"meeetings,\" and \"bidrectional\" in the Figure 1 caption; please proofread the manuscript.","section":"Throughout"},{"comment":"The notation in Eq. (11) is inconsistent: C is defined as a union of (i,j) index pairs from the inverted index, but the following sentence refers to a \"candidate frame c_k\" as though C contained frames; please clarify that candidates are indexed segments.","section":"II.C"},{"comment":"In Eq. (5), the symbol z_i appears in the cosine similarity, but z_t is the defined frame embedding; please correct the subscript and define the cosine similarity explicitly.","section":"II.B, Eq. (5)"},{"comment":"The EnCodec baseline is cited as \"[26]\" in Section III.C, but reference [26] is the S4 paper; EnCodec corresponds to reference [25] (Défossez et al.).","section":"III.C"},{"comment":"The Jaccard-consistency analysis in Table I measures the same token-consistency property that the training objective directly optimizes, so it is not an independent test of speaker invariance; consider reporting per-speaker-pair statistics or a separate speaker-discrimination evaluation to support the speaker-agnostic claim.","section":"IV.A"},{"comment":"Table II reports single numbers without significance tests or confidence intervals; given that each query set has only 300 terms, differences between close configurations (e.g., BEST-STD 256 vs. BEST-STD 512) may not be reliable; please add significance tests or confidence intervals, or clarify whether the numbers are averages over multiple runs.","section":"III.B and Table II"}],"recommendation":"major_revision","confidential_remarks":"The paper is a credible short-system paper with a clear method and a plausible empirical framework, but the central comparison currently rests on an under-specified retrieval protocol that must be corrected before the results can be assessed. The efficiency claim also needs direct evidence. Given that the code is public, these fixes are feasible within a revision; I do not see a fundamental flaw that would require rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sensible, clearly-written paper that combines bidirectional Mamba with vector quantization and a DTW-anchored contrastive loss to produce speech tokens for query-by-example spoken term detection. The combination is new, and the two-database evaluation shows large gaps over the baselines they chose. I'd take the numbers with a grain of salt until the retrieval protocol is pinned down.\n\nThe good bits: the method is easy to follow; the bidirectional Mamba is a direct extension of Mamba, and they show it beats a Transformer inside their own framework. The idea of indexing token bigrams with an inverted index is practical and moves STD beyond frame-level DTW. They are also transparent about failure cases (homophones) and about the MAP/MTWV trade-off.\n\nSoft spots, in decreasing order of importance. First, the retrieval protocol is under-specified. They say queries come from train-clean-100, which is also the archive, but they never say whether the query's source utterance or segment is removed from the index or from ground truth. If it isn't, every query has a guaranteed self-match that will rank first, which inflates MAP/MRR for every system. It's not necessarily fatal—if the task is to retrieve all occurrences, the source is a legitimate hit—but the paper has to say so. They also never define the segment hop h or what counts as a relevant segment, so the ground truth is ambiguous. Second, the headline efficiency claim ('significantly faster retrieval times') is unsupported: there are no timing measurements anywhere. Third, the token-consistency analysis (Table I) is, to some extent, measuring what the model was trained to optimize; comparing to HuBERT, WavLM, Encodec, and SpeechTokenizer, which optimize different objectives, isn't a fair comparison. Fourth, no significance tests or confidence intervals; some of the differences among their own configurations are small.\n\nThe stress-test note's self-match concern is real but should be framed as a protocol ambiguity rather than a fatal error. A careful re-evaluation with explicit exclusion rules and reported h and timings would settle it.\n\nBottom line: the core method is sound and the paper is worth sending to a serious referee, but it needs major revision on the evaluation before the main claims can be accepted.","headline":"A plausible new speech tokenization approach for spoken term detection, but the evaluation protocol is under-specified and the runtime claim is unmeasured; the method deserves peer review with major revision.","tokens_in":9867,"tokens_out":4808,"would_cite":false,"duration_ms":46341,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that query-by-example spoken term detection can be replaced by discrete, speaker-agnostic token sequences from a bidirectional Mamba encoder, enabling inverted-index retrieval that beats DTW-based baselines on LibriSpeech…","keywords":["spoken term detection","query-by-example","speech tokenization","bidirectional Mamba","state space models","self-supervised learning","inverted index","audio retrieval"],"falsifier":"A concrete check: corrupt the DTW alignments during training by adding random monotonic frame shifts and measure MAP and MRR on LibriSpeech; if retrieval accuracy does not drop, the DTW-alignment premise is not load-bearing.","tokens_in":8954,"feed_emoji":"🔍","tokens_out":10075,"duration_ms":89336,"temperature":0.7,"pith_summary":"Spoken term detection currently relies on either word-level embeddings that need segmentation or on expensive frame-level dynamic time warping. This paper claims that both can be sidestepped by encoding speech into discrete, speaker-agnostic semantic tokens, so that finding a spoken query becomes a text-style search over an inverted index. It introduces BEST-STD, a bidirectional Mamba encoder trained with a contrastive self-supervised loss that aligns different utterances of the same term frame-by-frame and quantizes the resulting embeddings. The reported result is higher retrieval accuracy than existing STD baselines on LibriSpeech and TIMIT, with substantially faster search than DTW-based methods. The practical payoff would be scalable voice search and audio archiving without an ASR system.","feed_headline":"Discrete speech tokens beat DTW for spoken-term search","feed_subtitle":"A bidirectional Mamba encoder turns speech into speaker-agnostic tokens that an inverted index can search.","key_machinery":"The load-bearing object is the bidirectional Mamba encoder: two identical Mamba state-space blocks process the input sequence forward and time-reversed, and their outputs are combined and projected into $\\ell^2$-normalized embeddings. A contrastive loss, using DTW to pick positive frame pairs between two utterances of the same word and random frames from other words as negatives, pushes the encoder to make embeddings speaker-invariant; a commitment loss pulls embeddings toward quantizer centroids. The discrete tokens are produced by nearest-centroid assignment, and retrieval rests on an inverted index of token bigrams scored by Jaccard similarity.","core_discovery":"The central discovery is that a bidirectional state-space encoder can be trained, without word labels, to emit token sequences that stay consistent across speakers and across renditions of the same term, and that those tokens are discriminative enough to make the detour through DTW unnecessary. The method learns frame-level contextual embeddings from log-Mel spectra using two Mamba blocks reading the input in opposite temporal directions, aligns pairs of same-word utterances with DTW to define anchor-positive contrastive pairs, and quantizes the embeddings with a vector quantizer updated by exponential moving averages. Retrieval then treats token sequences as text: it builds an inverted index over token bigrams and scores candidate segments by maximum Jaccard similarity. The paper reports that this pipeline consistently outperforms MFCC, phone-posterior, and bottleneck features matched with DTW, as well as tokenizers based on HuBERT, WavLM, SpeechTokenizer, and EnCodec, on both in-vocabulary and out-of-vocabulary query sets.","pith_inferences":["If the tokens are indeed speaker-agnostic subword units, the same encoder could be applied to out-of-domain audio or languages not seen in training, since no word lexicon or ASR is required.","The DTW-alignment premise could be stress-tested by replacing the aligner during training; if the method is robust, retrieval accuracy should degrade only slowly as alignments are corrupted.","The inverted-index scheme could be bolted onto any discrete speech tokenizer, meaning the speed gains are not necessarily tied to the Mamba encoder itself.","A retrieval-time disambiguation step using a small pronunciation or text lexicon could remove the homophone false positives without changing the tokenizer."],"forward_implications":["QbE-STD can be run with text-search data structures, so large audio archives can be searched much faster than with segmental DTW.","The tokenizer's consistency across speakers transfers to better retrieval for both in-vocabulary and out-of-vocabulary terms, because tokens represent subword units rather than whole words.","Bidirectional Mamba gives more useful temporal modeling for STD than a Transformer trained under the same framework.","Larger codebooks improve the MTWV retrieval score while slightly lowering MAP and MRR, so codebook size can be tuned to the application.","Homophones can be tokenized identically, creating false positives that a future extension would need to resolve."],"supporting_citations":[{"why":"Supplies the LibriSpeech training set, validation set, and search archive used in the main experiments.","marker":"[20]"},{"why":"Supplies the TIMIT database used as the second evaluation set for retrieval accuracy.","marker":"[21]"},{"why":"Provide the four state-of-the-art speech tokenizers (HuBERT, WavLM, SpeechTokenizer, EnCodec) whose discrete representations the method is compared against.","marker":"[22]–[25]"},{"why":"Defines the structured state-space model formulation that the bidirectional Mamba encoder builds on.","marker":"[26]"},{"why":"Provides the Mamba selective state-space model that the proposed bidirectional encoder is built from.","marker":"[29]"},{"why":"Provides the exponential-moving-average vector-quantization update used to learn the token codebook.","marker":"[30]"},{"why":"Motivates the $\\ell^2$-normalized centroids used to stabilize training and improve codebook use.","marker":"[31]"},{"why":"Defines the MAP and MRR metrics used to report retrieval performance.","marker":"[33]"},{"why":"Defines the MTWV metric used to report term-weighted retrieval performance.","marker":"[34]"}],"fun_headline_variants":["Speech tokens outpace DTW for term detection","Speaker-agnostic tokens make spoken-term search fast","Bidirectional Mamba speech tokens replace DTW search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The self-supervised training depends on DTW alignment between different utterances of the same word being correct at the frame level, so that the contrastive pairs it creates are truly the same sound; if the alignment is wrong, the tokens may become less speaker-invariant.","fun_headline_variants_meta":{"raw":{"variants":["Speech tokens outpace DTW for term detection","Speaker-agnostic tokens make spoken-term search fast","Bidirectional Mamba speech tokens replace DTW search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3246,"prompt_tokens":894,"completion_tokens":2352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2303}},"tokens_in":510,"tokens_out":2352,"duration_ms":16978,"temperature":1.0,"reasoning_tokens":2303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:31:57.539093+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: corrupt the DTW alignments during training by adding random monotonic frame shifts and measure MAP and MRR on LibriSpeech; if retrieval accuracy does not drop, the DTW-alignment premise is not load-bearing.","supporting_citations":[{"cited_title":"and Khudanpur, S., 2015, April","cited_arxiv_id":null,"evidence_quote":"Supplies the LibriSpeech training set, validation set, and search archive used in the main experiments."},{"cited_title":"and Pallett, D.S.,","cited_arxiv_id":null,"evidence_quote":"Supplies the TIMIT database used as the second evaluation set for retrieval accuracy."},{"cited_title":"and Vinyals, O., 2017","cited_arxiv_id":null,"evidence_quote":"Provides the exponential-moving-average vector-quantization update used to learn the token codebook."},{"cited_title":"Learning to rank for information retrieval","cited_arxiv_id":null,"evidence_quote":"Defines the MAP and MRR metrics used to report retrieval performance."},{"cited_title":"and Penagarikano, M., 2013","cited_arxiv_id":null,"evidence_quote":"Defines the MTWV metric used to report term-weighted retrieval performance."}],"review_version":1}