Pith. sign in

REVIEW 4 major objections 4 minor 43 references

Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single-speaker ASR model can transcribe one target speaker in overlapped multi-talker audio by injecting a learnable speaker kernel derived only from a binary speech-activity mask, with no speaker embeddings or…

desk verdict A genuinely new mask-conditioning mechanism for target-speaker ASR with strong headline numbers, but the full-overlap claim is untested and the lack of ablations makes the attribution of gains uncertain. read the letter →

arxiv 2506.22646 v2 pith:UOCGUGQP submitted 2025-06-27 eess.AS cs.SD

Weiqing Wang , Taejin Park , Ivan Medennikov , Jinhan Wang , Kunal Dhawan , He Huang , Nithin Rao Koluguri , Jagadeesh Balam
show 1 more author
Boris Ginsburg
This is my paper · ORCID
classification eess.AScs.SD
keywords multi-talkerASRtarget-speakerself-speakeradaptationspeakeractivitymaskFastConformerTransducerstreamingdiarizationcpWER
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that the difficult task of choosing which voice to transcribe in overlapping speech can be handled without speaker embeddings or enrollment audio. Its self-speaker adaptation method injects a small learned kernel, built from a per-frame speech-activity mask, into the encoder of an ordinary single-speaker ASR model; multi-talker transcription is then obtained by running one adapted instance per speaker. This matters because it turns multi-talker ASR into a small modification of a standard streaming ASR stack rather than a separate task requiring speaker profiles. The paper reports cpWER 2.2/2.8/5.0% on 1/2/3-mix LibriSpeechMix and 26.21% on real CH109 streaming conversations.

What carries the argument

The central mechanism is the speaker injection module: a small learnable function $f_{\text{inj}}(X_i, y_{\text{spk}}^k) = f_{\text{feedforward}}(X_i \odot y_{\text{spk}}^k)$ (two linear layers with an activation in between) inserted into the pre-encode layer of a FastConformer Transducer encoder, with a residual connection $X_{\text{inj}}^i = f_{\text{inj}}(X_i, y_{\text{spk}}^k) + X_i$. The binary speech-activity mask, a per-frame indicator of when the target speaker is speaking, multiplies the encoder features, and the learned feedforward network shapes those masked features into a speaker kernel that steers encoder states toward the target speaker. Multi-talker decoding then runs one model instance per speaker in a single batch.

What would settle it

Run the SSA model on fully overlapped two-speaker utterances where both speakers' speech-activity masks are identical (same onset, offset, and no solo segments); if cpWER degrades to near the no-mask baseline, the mask alone cannot carry speaker identity. A simpler check is to replace the mask with an all-ones vector at inference and see whether cpWER changes materially.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single-speaker ASR encoder can be turned into a target-speaker ASR system by injecting a learnable speaker kernel at the pre-encode layer, computed as $f_{\text{feedforward}}(X_i \odot y_{\text{spk}}^k)$ and added residually to the layer output. The only extra input is the target speaker's binary speech-activity mask $y_{\text{spk}}^k$; no speaker embedding, enrollment audio, or speaker identity is needed. Running one such adapted instance per speaker, with masks supplied by a diarization system, yields multi-talker transcription: cpWER 2.2/2.8/5.0% on 1/2/3-mix LibriSpeechMix and 26.21% on CH109 streaming, outperforming prior SOT, AFT-MT, and cascaded baselines.

Load-bearing premise

Everything rests on the assumption that a per-frame on/off speech-activity mask for the target speaker carries enough information about that speaker's voice to let the encoder separate and transcribe them, even in fully overlapped speech where the speakers' masks are nearly identical.

Editorial extensions

If this is right

  • Target-speaker ASR no longer needs enrollment audio or a pre-registered speaker profile; a diarization or personal-VAD mask is sufficient to select the speaker.
  • Existing single-speaker ASR stacks can be upgraded in place: the training objective, decoding procedure, and streaming caches stay the same, and only the injection module and multi-speaker training data are added.
  • Streaming multi-talker transcription is attainable at practical latencies, with cpWER 4.0/5.6% on 1/2-mix streaming LibriSpeechMix at 560 ms and 26.21% on real CH109 calls at 1120 ms.
  • Per-speaker decoding parallelizes naturally by batching one model instance per speaker, so added speakers add compute linearly without changing the algorithm.
  • The method also covers three-way overlap, with offline 3-mix cpWER of 5.0%, so the recipe is not limited to two-speaker conversations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The injection recipe is source-agnostic: if the mask acts as a soft pointer to one source, the same module could be applied to other separation tasks, though the paper only evaluates speech.
  • A natural stress test is to feed the model masks with degraded time resolution or injected speaker errors; the margin over cascaded baselines would reveal how much of the gain depends on perfect mask quality.
  • Per-speaker instances mean compute grows linearly with speaker count, so a single-pass multi-mask variant is an obvious efficiency follow-up not explored in the paper.
  • The reported t-SNE separation of encoder states under different kernels suggests the kernel works as a speaker-specific bias, pointing toward a hybrid with serialized output training that emits all speakers from one decoder.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes self-speaker adaptation (SSA), a target-speaker ASR approach that avoids enrollment audio and speaker embeddings. A learnable speaker kernel is computed from a binary speech-activity mask with a two-layer feedforward module and added to the pre-encode layer of a FastConformer Transducer; during multi-talker inference, one model instance is run per speaker using diarization masks from Sortformer. The authors report state-of-the-art cpWER on LibriSpeechMix offline (2.2%/2.8%/5.0% for 1/2/3-mix), streaming two-speaker results (4.0%/5.6% at 560 ms), and a CH109 streaming result of 26.21% that beats a same-backbone cascade and an offline LLM-based system.

Significance. If the reported results are robust, the paper makes a practically useful contribution: it removes the need for speaker enrollment or explicit speaker embeddings in target-speaker ASR, relies only on speech-activity masks, and extends naturally to streaming via multiple ASR instances. The method is parameter-efficient (1.1M injection parameters on top of public FastConformer and Sortformer checkpoints), and the training recipe is concrete enough to reproduce. The paper also includes a same-backbone cascaded baseline on CH109, which is a useful comparison. However, the contribution is empirical and currently lacks ablations, error bars, and overlap-ratio-stratified results, so the evidence that the SSA mechanism itself is responsible for the gains is not yet established.

major comments (4)
  1. [Section 2.1, Eqs. (1)-(2)] The sole speaker-specific input is the binary mask y_spk. If two speakers have identical activity patterns over an entire utterance, the two model instances receive the same X_i and the same mask, so the injected kernels are identical and no deterministic model with this architecture can emit different transcripts for the two speakers. Even in the less extreme one-speaker-fully-contained-in-another case, the contained speaker has no frames in which that speaker is active alone, so the mask provides no way to associate the overlapping mixture with the target voice. The paper does not report cpWER stratified by overlap ratio or by availability of target-only frames, and Section 3.1 does not describe the training simulation in enough detail to determine whether exact-full-overlap cases occur. Please add such a breakdown and discuss how the model behaves on the hardest overlap configurations that the abstract advertises.
  2. [Section 3.4, Tables 2-5] No variance estimates or repeated-seed results are reported. Several headline margins are very small (1-mix cpWER 2.2% vs 2.3% for MT-LLM; CH109 26.21% vs 26.31% for offline LLM-BSD), so the state-of-the-art claims are not statistically grounded. Please report error bars, at least across multiple random seeds, and indicate which differences are significant.
  3. [Section 3.4, Tables 2 and 5] There is no ablation that isolates the contribution of the SSA injection module. In particular, there is no same-backbone cascade baseline on the offline LibriSpeechMix sets, no fine-tuned FastConformer without injection, and no comparison using ground-truth instead of Sortformer masks. Because the method builds on strong public pretrained models (FastConformer and Sortformer), the reported gains could in principle come from the backbone and pretraining rather than from the proposed speaker kernel. Please add ablations that remove Eq. (2), change the injection layer, and vary the mask source.
  4. [Section 3.4.1, Table 4] The streaming three-speaker results are presented without any competing baseline, so the claim of strong streaming multi-talker performance is not comparative in the three-speaker case. Please add at least one streaming three-speaker baseline, for example an SOT or AFT system using the same FastConformer backbone and the same diarization model. The two-speaker streaming comparisons in Table 3 also use systems with different architectures and training recipes; a same-recipe baseline would materially strengthen the claim.
minor comments (4)
  1. [Section 2.1, Eq. (1)] The notation is inconsistent: 'Xi inj' should be X_i^{inj}, and 'yspkk' should be y_{spk}^k. Please clean up the equation formatting.
  2. [Figure 3] The t-SNE plot is only qualitative. Please add a quantitative measure, such as centroid distances or a small classification experiment, to support the claim that speaker injection induces separable encoder states.
  3. [Table 1] The table appears to contain stray axis labels from Figure 3 (the line '5 5 15 25 35 25 10 5'), which should be removed or placed in the figure caption.
  4. [Section 3.3] The cross-reference 'as mentioned in Section 3.3' for the streaming diarization model should probably be Section 3.2; please check all cross-references.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the SSA method is an empirical conditioning scheme, and no equation or fitted parameter reduces the reported results to the method's inputs.

full rationale

The paper's derivation chain is a standard supervised ASR recipe. Equations (1)-(2) define the speaker injection as X_i^inj = f_feedforward(X_i ⊙ y_spk) + X_i, where y_spk is a speech-activity mask supplied as an additional input. Nothing in the paper defines the target cpWER in terms of a fitted parameter, and the reported numbers are held-out evaluations on LibriSpeechMix and CH109 against external baselines. The training objective is the same single-speaker ASR objective, and the labels are the target speaker's reference transcriptions, so the 'prediction' is not an echo of the input mask. The only self-referential components are citations to NVIDIA models (FastConformer, streaming FastConformer, and Sortformer) used as the backbone and diarization source; these are publicly released checkpoints with independent benchmark results, and they function as tools rather than as premises that force the SSA outcome. There is also no imported uniqueness theorem or ansatz smuggled in by citation. A possible limitation is that two speakers with identical activity masks receive identical injected kernels on fully overlapped frames, which might limit worst-case overlap performance; however, that is an empirical robustness question, not a definitional reduction of the result to its inputs. Overall, the central claim has independent empirical content.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two borrowed NVIDIA backbones, FastConformer and Sortformer, rather than on any new physical entity; no graviton-style inventions appear. The free parameters are architecture and data-recipe choices that are not swept or ablated.

free parameters (3)
  • training data mixture ratio (1-mix:2-mix:3-mix) = 1:3:6 offline and for 3-speaker streaming; 1:9 for 2-speaker streaming
    Chosen in Section 3.4.1 without ablation; it directly determines the mixture conditions the model is optimized for.
  • speaker injection layer (pre-encode layer) = pre-encode layer of FastConformer encoder
    Section 2.1 and Figure 1 say 'selected' layers, but only pre-encode is tested; no ablation justifies this choice.
  • checkpoint averaging (top-5 by validation WER) = top five checkpoints averaged
    Sections 3.2 and 3.3 use a standard but manually chosen recipe that affects final performance; no sensitivity is reported.
assumptions (5)
  • domain assumption The streaming Sortformer diarization model produces speech-activity masks accurate enough for ASR conditioning at inference.
    Sections 2.3 and 3.2 use Sortformer to generate y_spk at inference while training uses ground-truth or frozen diarization labels; the paper reports DER but does not analyze how diarization errors propagate to ASR.
  • domain assumption A binary speech-activity mask y_spk alone is sufficient for the encoder to separate and transcribe an arbitrary target speaker in fully overlapped audio.
    Section 2.1, Eqs. 1-2: the only speaker-specific input is the mask; this is the core mechanism, and no ablation varies the amount of mask information.
  • domain assumption Models trained on LibriSpeechMix simulated mixtures transfer to real CH109 recordings.
    Section 3.1 and Table 5 evaluate on CH109 after finetuning on Fisher; no domain-gap analysis is provided beyond a single cpWER comparison.
  • domain assumption The public pretrained NVIDIA FastConformer checkpoints have no hidden exposure to the evaluation test sets.
    Section 3.2 initializes from publicly available models; the paper does not confirm that the evaluation sets were unseen during pretraining.
  • domain assumption The cpWER metric is applied consistently across the proposed system and all cited baselines.
    Section 3.1 defines cpWER following reference [29]; published baseline numbers are taken at face value without re-evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR." pith.science (2026). https://pith.science/paper/UOCGUGQP

@misc{pith2026250622646,
  author       = {Pith},
  title        = {Pith review of: Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UOCGUGQP}},
  note         = {Machine review of arXiv:2506.22646}
}
read the original abstract

We propose a self-speaker adaptation method for streaming multi-talker automatic speech recognition (ASR) that eliminates the need for explicit speaker queries. Unlike conventional approaches requiring target speaker embeddings or enrollment audio, our technique dynamically adapts individual ASR instances through speaker-wise speech activity prediction. The key innovation involves injecting speaker-specific kernels generated via speaker supervision activations into selected ASR encoder layers. This enables instantaneous speaker adaptation to target speakers while handling fully overlapped speech even in a streaming scenario. Experiments show state-of-the-art performance in both offline and streaming scenarios, demonstrating that our self-adaptive method effectively addresses severe speech overlap through streamlined speaker-focused recognition. The results validate the proposed self-speaker adaptation approach as a robust solution for multi-talker ASR under severe overlapping speech conditions.

Figures

Figures reproduced from arXiv: 2506.22646 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Multiple model instances are created and each model instance focuses on each speaker in the multi-talker recording. feature dimension, respectively. Here, finj can be any module that injects the speaker information (i.e., learnable speaker ker￾nel) into the layer output. In this paper, two linear layers with an activation in between were applied for simplicity: finj(X i , yspkk ) = ffeedforward(X i ⊙ yspkk ). (2) Du… view at source ↗
Figure 3
Figure 3. shows the t-SNE plot of the ASR encoder state (ac￾tivation at the last layer) for each token, where 2mix-spk0 and 2mix-spk1 are injected with the first and second speaker’s ker￾nel, respectively. It is important to note that these ASR encoder representations, 2mix-spk0 and 2mix-spk1, are derived from the same model and audio recording. We observe that the variabil￾ity introduced by speaker injection is smaller than … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 37 canonical work pages

  1. [1]

    Introduction Recent advancements in Automatic Speech Recognition (ASR), driven by improved architectures and larger training datasets, have significantly advanced the field. Concurrently, interest in multi-talker ASR has grown, particularly for applications such as analyzing natural conversations, developing voice as- sistants, and transcribing speech in ...

    work page Pith review arXiv 2025

  2. [2]

    However, the performance of such systems is highly dependent on the qual- ity of the provided queries

    Proposed Method Typically, target-speaker ASR systems rely on target speaker embeddings or enrollment audio to extract and utilize speaker- specific information from multi-talker utterances. However, the performance of such systems is highly dependent on the qual- ity of the provided queries. For instance, a clean and noise-free query audio is generally p...

    work page

  3. [3]

    there are the characters you can see

    Experiments and Results 3.1. Datasets and Evaluation Metrics The training dataset was simulated using the LibriSpeech Cor- pus [26]. Since the model was trained with a single-speaker objective, alignment was not required for timestamp generation during training. For evaluation, we utilized the LibriSpeechMix dataset [12], which includes 1-mix, 2-mix, and ...

    work page

  4. [4]

    Conclusion In this paper, we introduced a query-less speaker targeting ap- proach that employs a multi-instance encoder-decoder for each speaker. Following the design principle of maximizing the performance of the base monaural ASR system, the proposed multi-instance speaker targeting approach shows that the rela- tively long frame length can be addressed...

    work page

  5. [5]

    Learning hidden unit contri- butions for unsupervised speaker adaptation of neural network acoustic models,

    P. Swietojanski and S. Renals, “Learning hidden unit contri- butions for unsupervised speaker adaptation of neural network acoustic models,” in 2014 IEEE Spoken Language Technology Workshop (SLT), 2014, pp. 171–176

    work page 2014

  6. [6]

    Speaker adap- tation of neural network acoustic models using i-vectors,

    G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, “Speaker adap- tation of neural network acoustic models using i-vectors,” in2013 IEEE Workshop on Automatic Speech Recognition and Under- standing, 2013, pp. 55–59

    work page 2013

  7. [7]

    Front-end factor analysis for speaker verification,

    N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Trans- actions on Audio, Speech, and Language Processing , vol. 19, no. 4, pp. 788–798, 2010

    work page 2010

  8. [8]

    Fast speaker adaptation of hy- brid nn/hmm model for speech recognition based on discrimina- tive learning of speaker code,

    O. Abdel-Hamid and H. Jiang, “Fast speaker adaptation of hy- brid nn/hmm model for speech recognition based on discrimina- tive learning of speaker code,” in Proc. ICASSP, 2013, pp. 7942– 7946

    work page 2013

Show all 43 references
  1. [9]

    Speaker normalization using efficient fre- quency warping procedures,

    L. Lee and R. C. Rose, “Speaker normalization using efficient fre- quency warping procedures,” in Proc. ICASSP, vol. 1, 1996, pp. 353–356

    1996

  2. [10]

    The STC System for the CHiME-6 Challenge,

    I. Medennikov, M. Korenevsky, T. Prisyach, Y . Khokhlov, M. Ko- renevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. An- drusenko, I. Podluzhny et al., “The STC System for the CHiME-6 Challenge,” in CHiME 2020 Workshop on Speech Processing in Everyday Environments, 2020

    2020

  3. [11]

    Parameterised sigmoid and relu hidden activation functions for dnn acoustic modelling,

    C. Zhang and P. C. Woodland, “Parameterised sigmoid and relu hidden activation functions for dnn acoustic modelling,” in Six- teenth annual conference of the international speech communica- tion association, 2015

    2015

  4. [12]

    Low-rank plus diagonal adapta- tion for deep neural networks,

    Y . Zhao, J. Li, and Y . Gong, “Low-rank plus diagonal adapta- tion for deep neural networks,” inProc. ICASSP, 2016, pp. 5005– 5009

    2016

  5. [13]

    Speaker adaptation for wav2vec2 based dysarthric asr,

    M. K. Baskar, T. Herzig, D. Nguyen, M. Diez, T. Polzehl, L. Bur- get, J. ˇCernock`y et al., “Speaker adaptation for wav2vec2 based dysarthric asr,” arXiv preprint arXiv:2204.00770, 2022

    2022 arXiv

  6. [14]

    Confidence score based con- former speaker adaptation for speech recognition,

    J. Deng, X. Xie, T. Wang, M. Cui, B. Xue, Z. Jin, M. Geng, G. Li, X. Liu, and H. Meng, “Confidence score based con- former speaker adaptation for speech recognition,” in Proc. IN- TERSPEECH, 2022, pp. 2623–2627

    2022

  7. [15]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017

    2017 arXiv

  8. [16]

    Front- End Processing for the CHiME-5 Dinner Party Scenario,

    C. Boeddeker, J. Heitkaemper, J. Schmalenstroeer et al., “Front- End Processing for the CHiME-5 Dinner Party Scenario,” in CHiME5 Workshop, 2018

    2018

  9. [17]

    Serialized Output Training for End-to- End Overlapped Speech Recognition,

    N. Kanda, Y . Gaur et al., “Serialized Output Training for End-to- End Overlapped Speech Recognition,” in Proc. INTERSPEECH, 2020, pp. 2797–2801

    2020

  10. [18]

    End-to-End Monaural Multi-Speaker ASR System without Pretraining,

    X. Chang, Y . Qian et al., “End-to-End Monaural Multi-Speaker ASR System without Pretraining,” in Proc. ICASSP, 2019, pp. 6256–6260

    2019

  11. [19]

    MIMO-SPEECH: End-to-end Multi- Channel Multi-Speaker Speech Recognition,

    X. Chang, W. Zhang et al., “MIMO-SPEECH: End-to-end Multi- Channel Multi-Speaker Speech Recognition,” in Proc. ASRU , 2019, pp. 237–244

    2019

  12. [20]

    Alignment-free training for transducer-based multi-talker asr,

    T. Moriya, S. Horiguchi, M. Delcroix, R. Masumura, T. Ashihara, H. Sato, K. Matsuura, and M. Mimura, “Alignment-free training for transducer-based multi-talker asr,” arXiv preprint arXiv:2409.20301, 2024

    2024 arXiv

  13. [21]

    Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers,

    N. Kanda, Y . Gaur, X. Wang et al. , “Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers,” in Proc. INTERSPEECH, 2020

    2020

  14. [22]

    Serialized output train- ing by learned dominance,

    Y . Shi, L. Li, S. Yin, D. Wang, and J. Han, “Serialized output train- ing by learned dominance,” arXiv preprint arXiv:2407.03966 , 2024

    2024 arXiv

  15. [23]

    Sa-sot: Speaker- aware serialized output training for multi-talker asr,

    Z. Fan, L. Dong, J. Zhang, L. Lu, and Z. Ma, “Sa-sot: Speaker- aware serialized output training for multi-talker asr,” in Proc. ICASSP, 2024, pp. 9986–9990

    2024

  16. [24]

    End-to-end speaker-attributed asr with trans- former,

    N. Kanda, G. Ye, Y . Gaur, X. Wang, Z. Meng, Z. Chen, and T. Yoshioka, “End-to-end speaker-attributed asr with trans- former,” arXiv preprint arXiv:2104.02128, 2021

    2021 arXiv

  17. [25]

    Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Or- dering,

    I. Medennikov, T. Park, W. Wang, H. Huang, K. Dhawan, J. Wang, J. Balam, and B. Ginsburg, “Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Or- dering,” in Proc. INTERSPEECH, 2025

    2025

  18. [26]

    Di- cow: Diarization-conditioned whisper for target speaker au- tomatic speech recognition,

    A. Polok, D. Klement, M. Kocour, J. Han, F. Landini, B. Yusuf, M. Wiesner, S. Khudanpur, J. ˇCernock`y, and L. Burget, “Di- cow: Diarization-conditioned whisper for target speaker au- tomatic speech recognition,” Computer Speech & Language , vol. 95, p. 101841, 2026

    2026

  19. [27]

    Fast conformer with linearly scalable attention for efficient speech recognition,

    D. Rekesh, N. R. Koluguri, S. Kriman, S. Majumdar, V . Noroozi, H. Huang, O. Hrinchuk, K. Puvvada, A. Kumar, J. Balam et al., “Fast conformer with linearly scalable attention for efficient speech recognition,” in2023 IEEE Automatic Speech Recognition and Understanding Workshop...

    2023

  20. [28]

    Personal vad: Speaker-conditioned voice activity detection,

    S. Ding, Q. Wang, S.-Y . Chang, L. Wan, and I. Lopez Moreno, “Personal vad: Speaker-conditioned voice activity detection,” in The Speaker and Language Recognition Workshop (Odyssey 2020), 2020, pp. 433–439

    2020

  21. [29]

    Stateful conformer with cache-based inference for streaming au- tomatic speech recognition,

    V . Noroozi, S. Majumdar, A. Kumar, J. Balam, and B. Ginsburg, “Stateful conformer with cache-based inference for streaming au- tomatic speech recognition,” in Proc. ICASSP, 2024, pp. 12 041– 12 045

    2024

  22. [30]

    on the simulated training set derived from the LibriSpeech Corpus. The performance of the diarization model is reported in Table 1, which shows the Diarization Error Rate (DER) for LibriSpeechMix 2-mix, 3-mix with a collar of 0, and CH109 datasets with a collar of 0.25. For Li...

  23. [31]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in Proc. ICASSP, 2015, pp. 5206–5210

    2015

  24. [32]

    The Fisher Corpus: A Resource for the Next Generations of Speech-to-text,

    C. Cieri, D. Miller, and K. Walker, “The Fisher Corpus: A Resource for the Next Generations of Speech-to-text,” in Proc. LREC, 2004, pp. 69–71

    2004

  25. [33]

    Callhome american english speech,

    A. Canavan, D. Graff, and G. Zipperlen, “Callhome american english speech,” Web Download, Philadelphia, 1997, lDC97S42. [Online]. Available: https://catalog.ldc.upenn.edu/LDC97S42

    1997

  26. [34]

    CHiME-6 Chal- lenge: Tackling Multispeaker Speech Recognition for Unseg- mented Recordings,

    S. Watanabe, M. Mandel, J. Barker et al. , “CHiME-6 Chal- lenge: Tackling Multispeaker Speech Recognition for Unseg- mented Recordings,” in CHiME Workshop, 2020

    2020

  27. [35]

    Sortformer: Seamless integration of speaker diarization and asr by bridging timestamps and tokens,

    T. Park, I. Medennikov, K. Dhawan, W. Wang, H. Huang, N. R. Koluguri, K. C. Puvvada, J. Balam, and B. Ginsburg, “Sortformer: Seamless integration of speaker diarization and asr by bridging timestamps and tokens,” arXiv preprint arXiv:2409.06656, 2024

    2024 arXiv

  28. [36]

    SentencePiece: A Simple and Lan- guage Independent Subword Tokenizer and Detokenizer for Neu- ral Text Processing,

    T. Kudo and J. Richardson, “SentencePiece: A Simple and Lan- guage Independent Subword Tokenizer and Detokenizer for Neu- ral Text Processing,” inEMNLP: System Demonstrations, 2018

    2018

  29. [37]

    Decoupled weight decay regularization,

    I. Loshchilov, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  30. [38]

    A sidecar separator can convert a single-talker speech recognition system to a multi-talker one,

    L. Meng, J. Kang, M. Cui, Y . Wang, X. Wu, and H. Meng, “A sidecar separator can convert a single-talker speech recognition system to a multi-talker one,” in Proc. ICASSP, 2023, pp. 1–5

    2023

  31. [39]

    Empowering whisper as a joint multi-talker and target-talker speech recognition system,

    L. Meng, J. Kang, Y . Wang, Z. Jin, X. Wu, X. Liu, and H. Meng, “Empowering whisper as a joint multi-talker and target-talker speech recognition system,” in Proc. INTERSPEECH, 2024, pp. 4653–4657

    2024

  32. [40]

    Large language model can transcribe speech in multi-talker scenarios with versatile instructions,

    L. Meng, S. Hu, J. Kang, Z. Li, Y . Wang, W. Wu, X. Wu, X. Liu, and H. Meng, “Large language model can transcribe speech in multi-talker scenarios with versatile instructions,” arXiv preprint arXiv:2409.08596, 2024

    2024 arXiv

  33. [41]

    Streaming speaker-attributed asr with token-level speaker embeddings,

    N. Kanda, J. Wu, Y . Wu, X. Xiao, Z. Meng, X. Wang, Y . Gaur, Z. Chen, J. Li, and T. Yoshioka, “Streaming speaker-attributed asr with token-level speaker embeddings,” in Proc. INTERSPEECH, 2022, pp. 521–525

    2022

  34. [42]

    Self-supervised learning with bi- label masked speech prediction for streaming multi-talker speech recognition,

    Z. Huang, Z. Chen, N. Kanda, J. Wu, Y . Wang, J. Li, T. Yosh- ioka, X. Wang, and P. Wang, “Self-supervised learning with bi- label masked speech prediction for streaming multi-talker speech recognition,” in Proc. ICASSP, 2023, pp. 1–5

    2023

  35. [43]

    Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approach,

    T. J. Park, K. Dhawan, N. Koluguri, and J. Balam, “Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approach,” in Proc. ICASSP , 2024, pp. 10 861– 10 865

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.