Pith. sign in

REVIEW 4 major objections 7 minor 40 references

SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that explicitly conditioning an SOT decoder on speaker embeddings and speaker activity, both derived from a jointly trained EEND-EDA diarization branch, lowers word error rates on heavily overlapped speech without…

desk verdict A sensible SOT extension with an honest oracle control, but the conditioning claim is confounded by the new count-based decoding rule and the numbers need replication. read the letter →

arxiv 2506.12672 v1 pith:Q7F553OJ submitted 2025-06-15 cs.SD cs.CLeess.AS

Yuta Hirano , Sakriani Sakti This is my paper · ORCID
classification cs.SDcs.CLeess.AS
keywords overlappedspeechrecognitionserializedoutputtrainingspeakerdiarizationEEND-EDAdecoderconditioningembeddingsactivitymulti-talkerASR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an attention-based end-to-end speech recognizer can handle overlapping speakers better if it is explicitly told who is speaking and when, instead of relying on the decoder to separate speakers on its own. It proposes SC-SOT, which takes serialized output training (SOT) and augments the decoder with speaker embeddings and speaker activity labels produced by a jointly trained EEND-EDA diarization branch. On Libri2Mix and Libri3Mix, the speaker-embedding version with multi-task training lowers WER from 19.2/16.8 to 17.5/15.2 and from 36.0/34.5 to 33.2/30.9 relative to the SOT baseline. The paper also shows that combining activity conditioning with predicted embeddings hurts when diarization is imperfect, but recovers with oracle activity information, indicating that the diarization branch is the current bottleneck.

What carries the argument

The load-bearing object is the jointly trained EEND-EDA diarization branch, an end-to-end diarization model whose encoder-decoder attractor mechanism outputs per-speaker embeddings and per-frame activity posteriors. It feeds the SOT decoder in two ways: the current speaker's attractor embedding is injected into the feed-forward sublayer of the first decoder layer, and the activity posterior is used to subtract a penalty from source-target attention wherever the target speaker is predicted inactive. A third mechanism replaces end-of-sequence termination with decoding that stops only after the number of speaker-change tokens equals the speaker count predicted by the diarization branch. Together these turn the diarization output into explicit who-spoke-when conditioning for an attention-based decoder.

What would settle it

Substitute random vectors for the diarization branch's attractor embeddings at inference; if Libri2Mix WER still matches the reported 17.5/15.2, the improvement is not caused by speaker-identity information.

Watch

Extended reading notes

Core claim

The central claim is that the SOT decoder already performs implicit speaker separation, shown by attention visualizations in which overlapping frames are attended twice, but this separation is insufficient when acoustic cues are ambiguous. Conditioning the decoder on speaker embeddings from a jointly trained EEND-EDA model lets it focus on the target speaker's acoustic characteristics, and conditioning on speaker activity posteriors via an attention penalty helps suppress non-target speakers. In decoding, the speaker count predicted by EEND-EDA can also override the normal end-of-sequence termination. The best configuration uses speaker embeddings plus multi-task learning, with WER 17.5/15.2 on Libri2Mix and 33.2/30.9 on Libri3Mix; oracle activity information pushes Libri3Mix to 31.1/29.2, supporting the claim that imperfect diarization, not the conditioning idea, causes the combined variant to regress.

Load-bearing premise

The scheme rests on the jointly trained diarization branch remaining accurate and correctly ordered at inference; if its speaker embeddings or activity posteriors are wrong, conditioning can push WER above the unconditioned SOT baseline.

Editorial extensions

If this is right

  • Explicit speaker-embedding conditioning improves SOT word error rates on both two- and three-speaker mixtures without any enrollment utterances.
  • Because the speaker diarization branch is trained jointly with the ASR module, the conditioning signal costs no extra inference-time speaker enrollment.
  • Speaker activity conditioning is beneficial only when the diarization posteriors are accurate; replacing predicted activity with oracle activity lowers Libri3Mix WER from 36.2 to 31.1.
  • The best SC-SOT+MTL configuration raises speaker counting accuracy from 88.1% to 95.2% on Libri3Mix, showing the diarization branch's count can guide decoding.
  • Attention visualizations in the paper support the view that the SOT encoder preserves temporal order while the decoder performs implicit speaker separation, which is the behavior SC-SOT augments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the authors do not spell out is that a more accurate diarization branch should make the combined conditioning the best variant, since the oracle-activity result already marks that direction.
  • The conditioning recipe is not tied to SOT specifically: the same attractor-embedding injection and attention-penalty scheme could be applied to token-level serialized output training or other serialized multi-talker decoders, and the paper's reported gains suggest it is worth testing there.
  • The speaker-embedding gain may partly reflect the decoder receiving a stable token for the current speaker rather than detailed acoustic identity; replacing attractors with fixed per-speaker random vectors would separate those mechanisms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SC-SOT, an extension of serialized output training (SOT) for end-to-end multi-talker ASR. A jointly trained EEND-EDA diarization branch supplies speaker embeddings and per-frame speaker-activity posteriors; the decoder is conditioned on these signals via an FFN modification (Eq. 2) and an attention penalty (Eqs. 3-4). In addition, decoding is changed: the eos token is replaced by speaker-change tokens and termination is controlled by the diarization branch's predicted speaker count (Section 3.2.3). Experiments on Libri2Mix and Libri3Mix report WER improvements for the embedding-conditioned model (17.5/15.2 vs 19.2/16.8 on Libri2Mix test; 33.2/30.9 vs 36.0/34.5 on Libri3Mix test), while combining embedding and predicted activity conditioning degrades results (row 7); oracle activity recovers gains (row 8). The authors conclude that explicit speaker conditioning improves SOT.

Significance. If the results are uncontaminated, the paper makes a useful empirical contribution: it demonstrates a way to inject diarization-derived speaker information into an SOT decoder without enrollment, and it provides an attention-based behavioral probe of SOT. However, the central comparison is confounded by the simultaneous change of the decoding termination rule, so the attribution of the gains to conditioning is not established. The oracle-activity result shows potential, but not a deployable system. With the missing ablations, the paper's main claim as stated is not supported.

major comments (4)
  1. [Section 3.2.3 and Table 1] The SC-SOT rows (4-8) all incorporate the speaker-count decoding rule described in Section 3.2.3--replacing eos with sc and terminating only when the number of generated sc tokens equals the EEND-EDA-predicted speaker count--whereas the SOT baseline rows (2-3) use conventional eos termination. Consequently, the WER difference between row 2 and row 5 (16.8 to 15.2 on Libri2Mix test; 34.5 to 30.9 on Libri3Mix test) cannot be attributed solely to speaker-embedding conditioning; it may be wholly or partly due to the changed decoding rule. The same confound applies to the row 6 activity-only comparison. The paper needs an ablation that applies the same speaker-count decoding to the unconditioned SOT baseline (and to SOT+MTL), with no embedding or activity conditioning, to isolate the contribution of the conditioning signals.
  2. [Section 5 and Table 1, rows 7-8] The full proposed model with predicted speaker activity (row 7) is worse than the SOT baseline on Libri3Mix (36.2/34.6 vs 36.0/34.5 test WER) and only marginally better on Libri2Mix (18.7/16.2 vs 19.2/16.8), while the oracle-activity row (8) shows large gains (31.1/29.2 on Libri3Mix test). This means the combined system's benefit depends entirely on the oracle assumption; the paper should either train or improve the diarization branch to close this gap, or clearly frame the combined-conditioning result as a proof-of-concept rather than a system result. At minimum, the text should quantify how often the predicted activity differs from the oracle and how that correlates with WER.
  3. [Section 4.2] The model is trained with ground-truth diarization labels for conditioning but at inference uses the predicted branch outputs; this train/test mismatch is not discussed or tested. Since row 7's degradation is attributed to diarization errors, the paper should either train with predicted labels (e.g., through scheduled sampling or a differentiable approximation) or report an experiment that varies the quality of the activity input to show that the mismatch is not the cause.
  4. [Table 1 and Section 5] All numbers in Table 1 appear to be single runs with no variance, confidence intervals, or significance tests. Given that some reported differences are small (e.g., 15.6 vs 15.7 on Libri2Mix dev between rows 4 and 6), the 'consistently improves' language in Section 5 is stronger than the evidence supports. The authors should report multiple seeds or at least a significance measure for the main comparisons.
minor comments (7)
  1. [Section 5] The phrase 'without any additional inputs' is misleading because the model receives predicted speaker embeddings and activity from the diarization branch, which are additional inputs to the decoder; the authors likely mean 'without enrollment utterances,' but the wording should be corrected.
  2. [Section 3.1 and Figure 1] The attention visualization is qualitative and based on a single example; the claim that 'the decoder performs implicit speaker separation' needs supporting quantification or should be softened to reflect that it is an observed tendency rather than a proven mechanism.
  3. [Section 4.2] The hyperparameters alpha, c, and theta are fixed to single values with no sensitivity analysis; reporting the effect of c and theta on the activity-conditioning results would strengthen the paper.
  4. [Table 1 caption] The caption states 'WER, SCA and DER (dev / test)' but SCA and DER are missing for several rows (e.g., row 8 and the baseline SOT row); the authors should state why these metrics are omitted for those configurations.
  5. [Section 4.2] The phrase 'evaluation matrices' should be 'evaluation metrics.'
  6. [References] Reference [30] is incomplete (no booktitle or journal); it should be updated to a full citation.
  7. [Section 4.1] The statement that Libri2Mix and Libri3Mix were 'modified by introducing utterance start time offsets' should specify what offsets were used and why; the authors should also note whether the Libri3Mix simulation script will be released.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SC-SOT is evaluated on held-out test sets and its core WER claims are out-of-sample measurements, not recoveries of the training objective.

full rationale

No circular derivation chain was found. SC-SOT's two conditioning mechanisms are implemented as explicit architectural changes (Eq. 2 for speaker embeddings, Eqs. 3-4 for activity penalty) and are evaluated on Libri2Mix/Libri3Mix test splits that are not used for training, so the reported WER reductions (Table 1 rows 4-8 vs rows 1-3) are genuine out-of-sample measurements rather than consequences of how the model is defined. The EEND-EDA branch is jointly trained on ground-truth diarization labels, but this is standard supervised training, not fitting a result into the loss; row 8's oracle-conditioned experiment and the paper's own attribution of row 7's Libri3Mix regression to diarization error are openly reported limitations, not circular confirmations. The one methodological weakness is that every SC-SOT row includes the new speaker-count decoding rule of Section 3.2.3, so the marginal effect of the conditioning alone is not isolated; this is an experimental confound in attribution, not an equation-level reduction to the paper's inputs. The SCA numbers are also partly defined by that imposed stopping rule, but the paper transparently states the rule as its proposed mechanism rather than as an independent prediction. There are no load-bearing self-citations: the cited EEND-EDA, Transcribe-to-Diarize, and Polok et al. components are external prior work used as building blocks, not as proofs of the present claims. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The method relies on three hand-set hyperparameters and on domain assumptions about SOT ordering, the transfer of attention penalties, the sufficiency of WavLM features, and the representativeness of the attention visualization. The most fragile assumption is that the jointly trained diarization branch can provide accurate, correctly ordered speaker information at inference time.

free parameters (3)
  • attention penalty strength c = 50
    Hyperparameter in Equations 3-4 controlling how much non-target speaker frames are penalized; chosen by hand and not ablated in the paper.
  • speaker activity threshold theta = 0.5
    Threshold in Equation 4 for deciding whether a speaker is active at a time frame; chosen by hand with no reported sensitivity analysis.
  • diarization loss weight alpha = 0.1
    Weight balancing the ASR and diarization losses in Equation 1; fixed without a tuning study.
assumptions (4)
  • domain assumption SOT target sequences are serialized in FIFO order of utterance start times and this order matches the diarization branch's speaker indexing.
    Section 4.2 and Section 3.2.3 use FIFO order for permuting diarization labels and assume the s-th attractor corresponds to the s-th speaker in the serialized ASR label.
  • domain assumption Adding a fixed attention penalty to non-target frames in the decoder source-target attention transfers from Whisper-based target-speaker ASR to SOT-based multi-talker ASR.
    Section 3.2.2 adopts the method from Polok et al. [25] without an independent study of how SOT attention behaves under such penalties.
  • domain assumption WavLM weighted-sum features provide sufficient acoustic representation for both the ASR and diarization objectives.
    Section 4.2 fixes the input features to WavLM weighted sums; no comparison to other features or raw filter banks is provided.
  • ad hoc to paper The attention visualization in Figure 1 is representative evidence that the decoder performs implicit speaker separation.
    Section 3.1 bases the design motivation on qualitative attention plots without a quantitative separation metric or an error analysis across many utterances.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition." pith.science (2026). https://pith.science/paper/Q7F553OJ

@misc{pith2026250612672,
  author       = {Pith},
  title        = {Pith review of: SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q7F553OJ}},
  note         = {Machine review of arXiv:2506.12672}
}
read the original abstract

We propose Speaker-Conditioned Serialized Output Training (SC-SOT), an enhanced SOT-based training for E2E multi-talker ASR. We first probe how SOT handles overlapped speech, and we found the decoder performs implicit speaker separation. We hypothesize this implicit separation is often insufficient due to ambiguous acoustic cues in overlapping regions. To address this, SC-SOT explicitly conditions the decoder on speaker information, providing detailed information about "who spoke when". Specifically, we enhance the decoder by incorporating: (1) speaker embeddings, which allow the model to focus on the acoustic characteristics of the target speaker, and (2) speaker activity information, which guides the model to suppress non-target speakers. The speaker embeddings are derived from a jointly trained E2E speaker diarization model, mitigating the need for speaker enrollment. Experimental results demonstrate the effectiveness of our conditioning approach on overlapped speech.

Figures

Figures reproduced from arXiv: 2506.12672 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the conventional SOT-based end-to-end multi-talker ASR and the proposed approach. where Lasr and Ldiar are the loss function of SOT and EEND￾EDA, respectively. α is a hyperparameter to adjust the loss for speaker diarization task [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 25 canonical work pages

  1. [1]

    SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition

    Introduction Since the development of the first speech recognition system over half a century ago, speech recognition technology has steadily advanced. In particular, recent advances in automatic speech recognition (ASR) have been remarkable, largely due to the rise of deep learning [1, 2]. In common single-speaker tasks, especially, models surpassing hum...

    work page Pith review arXiv 2025

  2. [2]

    Related Works 2.1. Handling overlapped speech using speaker embed- dings Target speech extraction is a task that aims to isolate the speech of a specific speaker from a mixture containing multiple speak- ers, relying on cues about the target speaker’s identity. Speaker embedding is a common type of information used in target speech extraction [15, 16, 17]...

    work page

  3. [3]

    Proposed Approach 3.1. Encoder and decoder behavior in handling overlapped speech Although SOT-based models can transcribe overlapped speech without separating it, how SOT-based models achieve this has not been revealed. To understand SOT-based models’ behavior, we explored attention weights in the model. Figure 1 shows the examples of visualized attentio...

    work page

  4. [4]

    Dataset The training set consists of LibriSpeech-360h [27], Libri2Mix, and Libri3Mix [28]

    Experimental Setup 4.1. Dataset The training set consists of LibriSpeech-360h [27], Libri2Mix, and Libri3Mix [28]. Libri2Mix and Libri3Mix were modi- fied by introducing utterance start time offsets. For Libri2Mix, the simulating scripts and metadata provided by ESPnet [26] were used. The simulating scripts and metadata for generating the training set fro...

    work page

  5. [5]

    Our proposed SC-SOT models, particularly when combined with multi-task learning (MTL), demonstrate promis- ing performance gains over the conventional SOT-based model

    Experimental Results Table 1 presents the performance of our proposed SC-SOT- based model on the Libri2Mix and Libri3Mix datasets, along- side baseline comparisons and variations in conditioning con- figurations. Our proposed SC-SOT models, particularly when combined with multi-task learning (MTL), demonstrate promis- ing performance gains over the conven...

    work page

  6. [6]

    Our initial analy- sis, through visualization of source-target attention weights in an SOT-based model, revealed that the decoder performs a de- gree of speaker separation

    Conclusion This work investigated and enhanced the implicit speaker sep- aration capabilities of SOT-based models. Our initial analy- sis, through visualization of source-target attention weights in an SOT-based model, revealed that the decoder performs a de- gree of speaker separation. Building upon this key insight, we proposed an approach: conditioning...

    work page

  7. [7]

    Acknowledgements Part of this work is supported by JSPS KAKENHI Grant Num- bers JP21H05054 and JP23K21681

  8. [8]

    Sequence to multi-sequence learning via conditional chain mapping for mixture signals,

    J. Shi, X. Chang, P. Guo, S. Watanabe, Y . Fujita, J. Xu, B. Xu, and L. Xie, “Sequence to multi-sequence learning via conditional chain mapping for mixture signals,”Advances in Neural Informa- tion Processing Systems, vol. 33, pp. 3735–3747, 2020

    work page 2020

Show all 40 references
  1. [9]

    Recent advances in end-to-end automatic speech recognition,

    J. Li et al. , “Recent advances in end-to-end automatic speech recognition,” APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1, 2022

    2022

  2. [10]

    End-to-end speech recognition: A survey,

    R. Prabhavalkar, T. Hori, T. N. Sainath, R. Schl¨uter, and S. Watan- abe, “End-to-end speech recognition: A survey,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023

    2023

  3. [11]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  4. [12]

    Anatomy of industrial scale multilingual asr,

    F. M. Ramirez, L. Chkhetiani, A. Ehrenberg, R. McHardy, R. Botros, Y . Khare, A. Vanzo, T. Peyash, G. Oexle, M. Liang et al. , “Anatomy of industrial scale multilingual asr,” arXiv preprint arXiv:2404.09841, 2024

    2024 arXiv

  5. [13]

    Less is more: Accurate speech recog- nition & translation without web-scale data,

    K. C. Puvvada, P. ˙Zelasko, H. Huang, O. Hrinchuk, N. R. Koluguri, K. Dhawan, S. Majumdar, E. Rastorgueva, Z. Chen, V . Lavrukhin et al. , “Less is more: Accurate speech recog- nition & translation without web-scale data,” arXiv preprint arXiv:2406.19674, 2024

    2024 arXiv

  6. [14]

    Recognizing multi-talker speech with permutation invariant training,

    D. Yu, X. Chang, and Y . Qian, “Recognizing multi-talker speech with permutation invariant training,” arXiv preprint arXiv:1704.01985, 2017

    2017 arXiv

  7. [15]

    Seri- alized output training for end-to-end overlapped speech recogni- tion,

    N. Kanda, Y . Gaur, X. Wang, Z. Meng, and T. Yoshioka, “Seri- alized output training for end-to-end overlapped speech recogni- tion,” arXiv preprint arXiv:2003.12687, 2020

    2003 arXiv

  8. [16]

    Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures,

    K. ˇZmol´ıkov´a, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. ˇCernock`y, “Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures,” IEEE Journal of Selected Topics in Signal Processing , vol. 13, no. 4, pp. 800–814, 2019

    2019

  9. [17]

    Streaming multi-talker asr with token-level serialized output training,

    N. Kanda, J. Wu, Y . Wu, X. Xiao, Z. Meng, X. Wang, Y . Gaur, Z. Chen, J. Li, and T. Yoshioka, “Streaming multi-talker asr with token-level serialized output training,” arXiv preprint arXiv:2202.00842, 2022

    2022 arXiv

  10. [18]

    Surt 2.0: Advances in transducer-based multi-talker speech recognition,

    D. Raj, D. Povey, and S. Khudanpur, “Surt 2.0: Advances in transducer-based multi-talker speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023

    2023

  11. [19]

    Permutation invari- ant training of deep models for speaker-independent multi-talker speech separation,

    D. Yu, M. Kolbæk, Z.-H. Tan, and J. Jensen, “Permutation invari- ant training of deep models for speaker-independent multi-talker speech separation,” in IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), 2017, pp. 241–245

    2017

  12. [20]

    A purely end-to-end system for multi-speaker speech recognition,

    H. Seki, T. Hori, S. Watanabe, J. L. Roux, and J. R. Hershey, “A purely end-to-end system for multi-speaker speech recognition,” arXiv preprint arXiv:1805.05826, 2018

    2018 arXiv

  13. [21]

    Mimo-speech: End-to-end multi-channel multi-speaker speech recognition,

    X. Chang, W. Zhang, Y . Qian, J. Le Roux, and S. Watanabe, “Mimo-speech: End-to-end multi-channel multi-speaker speech recognition,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 237–244

    2019

  14. [22]

    End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,

    S. Horiguchi, Y . Fujita, S. Watanabe, Y . Xue, and K. Naga- matsu, “End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,”arXiv preprint arXiv:2005.09921, 2020

    2005 arXiv

  15. [23]

    Neural target speech extraction: An overview,

    K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. ˇCernock`y, and D. Yu, “Neural target speech extraction: An overview,”IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8–29, 2023

    2023

  16. [24]

    Transcribe-to-diarize: Neural speaker diariza- tion for unlimited number of speakers using end-to-end speaker- attributed asr,

    N. Kanda, X. Xiao, Y . Gaur, X. Wang, Z. Meng, Z. Chen, and T. Yoshioka, “Transcribe-to-diarize: Neural speaker diariza- tion for unlimited number of speakers using end-to-end speaker- attributed asr,” in IEEE International Conference on Acoustics, Speech and Signal Processing...

    2022

  17. [25]

    Adaptive blind audio source extraction supervised by dominant speaker identification using x-vectors,

    J. Jansk `y, J. M´alek, J. ˇCmejla, T. Kounovsk`y, Z. Koldovsk`y, and J. ˇZd’´ansk`y, “Adaptive blind audio source extraction supervised by dominant speaker identification using x-vectors,” in IEEE In- ternational Conference on Acoustics, Speech and Signal Process- ing (ICASSP...

    2020

  18. [26]

    Target-speaker voice activity de- tection: a novel approach for multi-speaker diarization in a dinner party scenario,

    I. Medennikov, M. Korenevsky, T. Prisyach, Y . Khokhlov, M. Ko- renevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. An- drusenko, I. Podluzhny et al., “Target-speaker voice activity de- tection: a novel approach for multi-speaker diarization in a dinner party scenario,” ar...

    2005 arXiv

  19. [27]

    Tar- get speaker voice activity detection with transformers and its in- tegration with end-to-end neural diarization,

    D. Wang, X. Xiao, N. Kanda, T. Yoshioka, and J. Wu, “Tar- get speaker voice activity detection with transformers and its in- tegration with end-to-end neural diarization,” in IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  20. [28]

    Adapting self-supervised models to multi-talker speech recognition using speaker embeddings,

    Z. Huang, D. Raj, P. Garc ´ıa, and S. Khudanpur, “Adapting self-supervised models to multi-talker speech recognition using speaker embeddings,” inIEEE international conference on acous- tics, speech and signal processing (ICASSP), 2023, pp. 1–5

    2023

  21. [29]

    The Conformer encoder has 12 layers, while the Transformer decoder is structured with 6 layers

    and a Transformer decoder [30]. The Conformer encoder has 12 layers, while the Transformer decoder is structured with 6 layers. Both the encoder and decoder utilize 4 attention heads and a hidden dimension of 256 and a 2048-dimensional FFN. The kernel size of the convolution l...

  22. [30]

    Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376

    2006

  23. [31]

    Target speaker asr with whisper,

    A. Polok, D. Klement, M. Wiesner, S. Khudanpur, J. ˇCernock`y, and L. Burget, “Target speaker asr with whisper,” arXiv preprint arXiv:2409.09543, 2024

    2024 arXiv

  24. [32]

    Di- cow: Diarization-conditioned whisper for target speaker au- tomatic speech recognition,

    A. Polok, D. Klement, M. Kocour, J. Han, F. Landini, B. Yusuf, M. Wiesner, S. Khudanpur, J. ˇCernock`y, and L. Burget, “Di- cow: Diarization-conditioned whisper for target speaker au- tomatic speech recognition,” arXiv preprint arXiv:2501.00114 , 2024

    2024 arXiv

  25. [33]

    But/jhu system descrip- tion for chime-8 notsofar-1 challenge,

    A. Polok, D. Klement, J. Han, ˇS. Sedl ´aˇcek, B. Yusuf, M. Ma- ciejewski, M. S. Wiesner, and L. Burget, “But/jhu system descrip- tion for chime-8 notsofar-1 challenge,” inProc. CHiME 2024, pp. 18–22

    2024

  26. [34]

    Espnet: End-to-end speech processing toolkit,

    S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. E. Y . Soplin, J. Heymann, M. Wiesner, N. Chen et al. , “Espnet: End-to-end speech processing toolkit,” arXiv preprint arXiv:1804.00015, 2018

    2018 arXiv

  27. [35]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in IEEE international conference on acoustics, speech and signal processing (ICASSP), 2015, pp. 5206–5210

    2015

  28. [36]

    Librimix: An open-source dataset for generalizable speech separation,

    J. Cosentino, M. Pariente, S. Cornell, A. Deleforge, and E. Vin- cent, “Librimix: An open-source dataset for generalizable speech separation,” arXiv preprint arXiv:2005.11262, 2020

    2005 arXiv

  29. [37]

    Conformer: Convolution- augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu et al. , “Conformer: Convolution- augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  30. [38]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural In- formation Processing Systems, 2017

    2017

  31. [39]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  32. [40]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.