{"id":"05673d42-4356-4fc0-aaba-978d91791d25","arxiv_id":"2501.00114","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Diarization-conditioned Whisper with frame-level transforms and query-key biasing achieves strong target-speaker ASR on AMI, NOTSOFAR-1, Libri2Mix, and LibriCSS.","lead":"DiCoW adapts Whisper for target-speaker speech recognition by conditioning it on speaker diarization labels instead of speaker embeddings. It reports strong results on meeting corpora and synthetic mixtures, with code released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle-to-automatic diarization transfer is the load-bearing assumption; the 19.7-to-33.5 tcpWER drop on NOTSOFAR-1 shows DiCoW has no learned recovery from real diarization errors, so the real-world TS-ASR claim is not yet supported.","rationale":"The paper's central advance is that frame-level diarization labels can replace speaker embeddings as the conditioning signal for TS-ASR. The oracle-conditioned results are strong and the Branchformer experiment (Section 6.6) shows the method is not Whisper-specific, which is real evidence in favor of the conceptual claim. However, the advertised use case is real-world multi-speaker recordings, and Table 6 is the paper's own stress test: automatic diarization raises tcpWER on every dataset, most severely on NOTSOFAR-1 (19.7 to 33.5). The paper's explanation in Section 6.3—training only on hard oracle labels plus FDDT's suppressive initialization—identifies exactly why there is no recovery mechanism for diarization misses and confusions. This matches the reader's weakest assumption. I also considered two other concerns: the Table 3 state-of-the-art comparisons are not protocol-matched, and Table 9 shows single-speaker degradation on TED-LIUM and VoxPopuli when the CTC head is used. Both are real and should be addressed in revision, but they are secondary to the oracle-to-real transfer issue: the conceptual claim 'diarization can replace embeddings' is only practically true if the system tolerates the diarizer's actual errors. The proposed synthetic-corruption or DiariZen-label training test would settle whether the gap is a fixable training-distribution artifact or an intrinsic hard-STNO limitation. Since the paper already acknowledges the diarization dependency in its limitations and the issue is correctable rather than fatal, CONDITIONAL remains the appropriate verdict.","tokens_in":22216,"tokens_out":7346,"duration_ms":69748,"concrete_test":"Retrain the FDDT multi-domain model on AMI and NOTSOFAR using DiariZen hard labels on their training splits (or oracle labels corrupted to match the miss/false-alarm/confusion rates in Table 7), then evaluate on NOTSOFAR-1 eval-small with DiariZen labels. If tcpWER drops substantially from 33.5% toward the 19.7% oracle result, the gap is a training-distribution mismatch and is fixable by training with realistic diarization errors; if tcpWER remains near 33.5%, the limitation is intrinsic to conditioning on imperfect hard STNO masks, and the real-world claim in the abstract should be weakened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DiCoW's central practical claim—that diarization outputs can replace speaker embeddings for target-speaker ASR in real recordings—rests on transfer from oracle hard STNO masks at training to DiariZen hard masks at inference. The model is trained only on ground-truth hard labels (Section 5.3) and decoded with hard STNO decisions (Section 6.3), so it never sees miss, false-alarm, or confusion errors. Table 6 quantifies the consequence: on NOTSOFAR-1 eval-small, tcpWER rises from 19.7% with oracle labels to 33.5% with DiariZen labels, and cpWER from 17.2% to 23.6%. Table 7 shows the diarizer has 22.0% DER on NOTSOFAR, with 7.0% miss and 8.8% confusion. Section 6.3 states that the system 'has not encountered such cases during training' and that FDDT's suppressive initialization (Section 4.4) makes silence frames ignored from the start, so missed speech is largely unrecoverable. The headline comparisons in Table 3 are reported for oracle diarization, while the automatic-diarization rows degrade substantially; therefore the practical 'diarization-conditioned' claim is not yet established without a demonstrated recovery mechanism for realistic diarization errors.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes DiCoW, an adaptation of Whisper for target-speaker ASR in which frame-level speaker diarization probabilities are converted into Silence/Target/Non-target/Overlap (STNO) masks. Two conditioning mechanisms are presented: query-key biasing (QKb), which biases attention scores away from non-target frames via an extended query/key formulation, and frame-level diarization-dependent transformations (FDDT), which apply class-specific affine transforms to encoder hidden states. A CTC head with joint CTC/attention decoding and a Co-Attention module for multi-speaker interaction are also studied. Experiments fine-tune Whisper-large-v3-turbo on AMI, NOTSOFAR-1, and Libri2Mix and evaluate on these plus LibriCSS, comparing oracle and automatic diarization, with a Branchformer experiment for generality. The paper reports state-of-the-art or competitive target-speaker WERs under oracle diarization, along with a substantial degradation when automatic diarization is used, and concludes that the method maintains single-speaker performance.","tokens_in":22432,"tokens_out":8738,"duration_ms":83160,"significance":"If the results are reproducible, the idea of conditioning on diarization outputs rather than speaker embeddings is a useful and economical direction for TS-ASR, with several praiseworthy elements: the code is released; the evaluation spans real (AMI, NOTSOFAR-1) and synthetic corpora; the comparison between oracle and system diarization is presented explicitly; and the Branchformer experiment supports transferability beyond Whisper. The paper is also unusually honest about limitations. However, several headline claims go beyond what the experiments establish, particularly regarding real-diarization robustness, the 'best results to date' statement, and the preservation of single-speaker accuracy.","major_comments":[{"comment":"The central claim of real-world applicability is not established for the automatic-diarization setting. The model is trained only on oracle hard STNO labels (Section 5.3) and decoded with hard DiariZen labels (Section 5.4 and Section 6.3), so it has never seen miss, false-alarm, or confusion errors; Section 6.3 explicitly states that the system 'has not encountered such cases during training' and that the FDDT suppressive initialization makes missed speech unrecoverable. Table 6 shows tcpWER on NOTSOFAR-1 eval-small rising from 19.7% with oracle labels to 33.5% with DiariZen labels, with tcORC-WER rising from 19.1% to 22.6%, and LibriCSS worsening from 8.8% to 11.0% tcpWER. Because the abstract and introduction claim 'more reliable transcription in real-world multi-speaker recordings' and 'even when automatic diarization is used' (Section 1), the authors should either add a training scheme with simulated diarization errors or soft labels, or substantially temper these claims to oracle-conditioned performance.","section":"§6.3, Tables 6–7"},{"comment":"The 'best results to date' claims are not supported by the comparison as presented. Table 3 selects a different system variant for different datasets (MD FDDT for AMI/NOTSOFAR-1/LibriCSS and SD+Co-Attention for Libri2Mix), combines oracle and system-diarization rows, uses different metrics per dataset (cpWER for AMI, tcpWER/tcORC-WER for NOTSOFAR-1, cpWER for Libri2Mix/LibriCSS), includes baseline numbers marked with a dagger that are not directly comparable because they use utterance-group scoring, and reports some ORC-WER values as collar-inflated approximations marked with a star. Under these conditions, the statement that 'The proposed system also achieves the best results to date on the Libri2Mix and LibriCSS datasets for both real- and ground-truth diarization' overreaches. Please provide a consistent protocol, or explicitly restrict the claim to the configurations and metrics that are directly comparable.","section":"§6.1, Table 3"},{"comment":"The abstract's claim that DiCoW maintains Whisper's accuracy and robustness on single-speaker data is stronger than the evidence. With the proposed model and lambda=0.2, TED-LIUM WER is 7.8% versus 4.3% for Whisper with beam size 5, and VoxPopuli is 11.2% versus 10.0%; even without CTC rescoring (lambda=0.0) the model gives 5.0% versus 4.3% and 11.0% versus 10.0%. The conclusion's phrase 'does not substantially degrade' is defensible, but the abstract should be qualified, for example by saying that accuracy is largely preserved on LibriSpeech while some degradation occurs on out-of-domain single-speaker sets, or the authors should provide a model variant that better preserves generalization.","section":"§6.5, Table 9"}],"minor_comments":[{"comment":"The description of lambda=1.0 as autoregressive decoding with the top 1000 tokens rescored by CTC contradicts Eq. (3), where lambda=1.0 would mean pure CTC decoding; please clarify the actual decoding objective or rename the parameter.","section":"§6.4, Table 8"},{"comment":"The bias constant c is introduced as a nonnegative value but the non-target score is -c; please define the sign convention explicitly and state whether the same c is used for all layers and heads and how it was selected.","section":"§4.3, Eq. (10)"},{"comment":"The superscript and subscript notation for layer indices is inconsistent (z_l^t versus hat-z_t^l), which makes the FDDT equations harder to follow; a single consistent indexing convention would improve readability.","section":"§4.4, Eq. (13)"},{"comment":"The spelling 'Librispeech' appears alongside 'LibriSpeech' in the same section, and the description of the CTC preheat stage could state more precisely that monitoring is done on LibriSpeech dev-clean and dev-other while training uses the 960h training set.","section":"§5.3"},{"comment":"For the starred ORC-WER approximations obtained by increasing the time collar, the table should state the collar value used for each entry, since a very large collar can make the approximation too loose for meaningful comparison with published ORC-WER numbers.","section":"§6.1, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper shares substantial components with the authors' own prior work arXiv:2409.09543, cited as [35], including FDDT and the suppressive initialization. The manuscript should state explicitly what is new in DiCoW relative to [35], because as written a reader cannot tell which parts are novel contributions of this submission. I would also ask the editors to verify whether any of the Table 3 comparisons were produced by the authors' earlier system variant. This does not change my technical verdict but should be addressed editorially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competently done empirical paper showing you can turn Whisper into a target-speaker ASR system by conditioning on diarization labels instead of speaker embeddings. The genuinely new pieces are QK biasing and the STNO-mask conditioning; FDDT comes from their own prior work, and they say so. Code and recipes are public. The experiments span four datasets, include real diarization, and extend to Branchformer. That is a solid contribution.\n\nThe soft spots are real but not fatal. First, the 'best results to date' claims in Table 3 rely on picking the best variant per dataset — MD FDDT for some, SD+Co-Attention for Libri2Mix — and the comparisons are not always protocol-matched. That weakens the SOTA narrative. Second, the abstract says single-speaker performance is maintained, but Table 9 shows clear degradation on TED-LIUM and VoxPopuli, especially with the CTC head. The paper admits 'slightly degrades its domain generalization capabilities', which is more accurate. Third, and most importantly: the model is trained only on ground-truth hard diarization labels, while inference uses DiariZen hard labels. On NOTSOFAR-1, tcpWER rises from 19.7% with oracle labels to 33.5% with real diarization. The paper is transparent about this — Sections 6.3 and 7 both flag it — but it means the 'real-world' claim is only partially supported. The model has no learned recovery from diarization misses or confusions.\n\nI would not call this a fatal flaw because the oracle-label results are strong, the real-diarization numbers are still competitive with several published systems, and the gap is quantified rather than hidden. But the authors should either soften the abstract or add robustness training (e.g., simulated diarization errors) before claiming real-world readiness.\n\nWho is this for? Anyone working on TS-ASR or meeting transcription. It is a useful proof that diarization conditioning can replace speaker embeddings, and the code release makes it easy to build on. I would send it to review. I would not cite it in my own next paper unless I needed the QKb idea specifically, but that is a personal call.","headline":"Diarization-conditioned Whisper for TS-ASR is a genuine contribution, but the paper overstates real-world robustness and cherry-picks per-dataset variants.","tokens_in":23099,"tokens_out":4494,"would_cite":false,"duration_ms":42300,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that conditioning Whisper on frame-level diarization labels—silence, target, non-target, overlap—rather than speaker embeddings, produces a target-speaker ASR system that matches or beats previous methods on AMI…","keywords":["Diarization-Conditioned Whisper","Target-Speaker ASR","Speaker Diarization","STNO masks","Whisper Adaptation","Frame-Level Diarization-Dependent Transformations","Query-Key Biasing","Multi-Speaker ASR"],"falsifier":"Take the released code, run inference on the reported test sets while randomly corrupting oracle diarization labels at rates from 5% to 30% (missed speech and speaker confusions), and check whether tcpWER rises in proportion to the corruption; the paper's assumption predicts a sharp rise with no recovery of missed segments.","tokens_in":21954,"feed_emoji":"🎙️","tokens_out":14126,"duration_ms":112559,"temperature":0.7,"pith_summary":"DiCoW tries to make a pre-trained single-speaker ASR model do target-speaker transcription by conditioning on who is speaking when, instead of on what the target speaker sounds like. For each target speaker, diarization outputs are collapsed into a four-way per-frame label — silence, target-only, non-target, and overlap — and injected into Whisper through attention biasing (QKb) and per-layer frame-wise affine transformations (FDDT). The paper reports that this matches or beats prior speaker-embedding and enrollment-based systems on AMI, NOTSOFAR-1, Libri2Mix, and LibriCSS, while leaving single-speaker performance close to the original Whisper. If this holds, meeting transcription becomes simpler and more robust to unseen speakers, because no component has to learn a mapping from speaker-embedding space to ASR space.","feed_headline":"Diarization masks turn Whisper into a target-speaker transcriber","feed_subtitle":"Frame-level speaker-activity labels give competitive meeting transcription without enrollment speech or speaker embeddings.","key_machinery":"The load-bearing object is the STNO mask: for a target speaker $s_k$, each frame $t$ is assigned a probability vector $M_t = (p^S_t,\\ p^T_t,\\ p^N_t,\\ p^O_t)^\\top$ whose entries are the probabilities of silence, target-only, non-target, and overlap, computed from the diarization matrix $D\\in[0,1]^{S\\times T}$. Two mechanisms carry the conditioning: QKb extends the attention query and key with a constant that initially subtracts a bias $c$ from non-target frames' attention scores, and FDDT replaces each encoder layer's input frame $z^l_t$ with a convex combination of four trainable affine maps $W^l_c z^l_t + b^l_c$ weighted by the STNO probabilities. The suppressive initialization (zeroing $W^l_S$ and $W^l_N$, identity for target and overlap) keeps the pre-trained model intact at the start of fine-tuning. A Co-Attention module lets the per-speaker decoding channels exchange information and resolve which instance decodes overlapping speech.","core_discovery":"The paper's central claim is that target-speaker ASR can be driven by diarization activity labels instead of speaker embeddings. For each target speaker, DiCoW converts diarization probabilities into a fixed-size STNO mask—per-frame probabilities of silence, target speaker only, non-target speaker(s), and overlap—and feeds this mask into Whisper's encoder via frame-level diarization-dependent transformations (FDDT) and query-key biasing (QKb). With ground-truth diarization, the authors report 17.2 cpWER on AMI-sdm, 19.7 tcpWER on NOTSOFAR-1 eval-small, 4.4 cpWER on Libri2Mix test-clean, and 5.6 cpWER on LibriCSS test; with automatic diarization from the DiariZen system, the corresponding numbers are 23.6, 33.5, 6.0, and 8.5. They state that DiCoW achieves the best results to date on Libri2Mix and LibriCSS for both real and oracle diarization, and that the same FDDT mechanism transfers to a non-Whisper Branchformer model.","pith_inferences":["A natural extension the paper leaves implicit is to train on soft STNO probabilities or to corrupt oracle labels with miss and confusion noise during training; the 13.8-point tcpWER drop on NOTSOFAR-1 suggests that hard-label-only training is the main recoverable loss, and we would expect randomized label corruption to close most of it.","Because FDDT is essentially a per-frame, per-class affine mixture, it could be reused as a generic class-conditional conditioning layer for other frame-labeled speech tasks (e.g., language ID, emotion, or source-type conditioning), not just speaker activity.","The Co-Attention module's gains appeared mainly on Libri2Mix, where speech is fully overlapped; this suggests that future work on high-overlap meetings should scale up speaker-interaction layers, while for typical meeting overlap rates the per-speaker independent decoders are already near their ceiling.","If the oracle-label results hold across more languages and acoustic conditions, the method could make speaker-attributed transcription a post-processing step on top of any off-the-shelf ASR, since the only external requirement is a diarizer that outputs per-frame activity."],"forward_implications":["Meeting transcription can be organized as one diarization pass followed by parallel Whisper decoders per speaker, with no enrollment speech and no speaker-embedding conditioning.","Systems trained this way inherit Whisper's single-speaker robustness: on LibriSpeech, TED-LIUM, and VoxPopuli the fine-tuned model stays within about one absolute WER point of unmodified Whisper.","Because diarization activity is a generic frame-level signal, the same STNO injection should apply to any pre-trained encoder-decoder ASR; the paper's Branchformer experiment supports this.","Adding the CTC head and hybrid CTC/attention decoding improves target-speaker accuracy on NOTSOFAR-1 beyond what the Whisper decoder alone achieves, suggesting the conditioning and alignment benefits are additive.","The gap between oracle and automatic diarization (e.g. 19.7 to 33.5 tcpWER on NOTSOFAR-1) becomes the main remaining cost, making diarization quality the bottleneck rather than ASR."],"supporting_citations":[{"why":"Supplies the pre-trained Whisper backbone that DiCoW conditions on diarization labels.","marker":"[15]"},{"why":"Defines NOTSOFAR-1, one of the real-world evaluation benchmarks and its challenge protocol.","marker":"[16]"},{"why":"Provides the AMI meeting corpus used for training and evaluation.","marker":"[17]"},{"why":"Provides the Libri2Mix synthetic mixture benchmark for comparison with prior methods.","marker":"[18]"},{"why":"Provides the LibriCSS overlapping-speech benchmark used for cross-dataset evaluation.","marker":"[21]"},{"why":"The DiariZen diarization system whose hard labels drive inference-time conditioning.","marker":"[42]"},{"why":"A prior target-speaker Whisper method that conditions on a speaker embedding, the approach DiCoW aims to replace.","marker":"[25]"},{"why":"A Whisper-based multi-talker and target-talker baseline using activation speech rather than embeddings.","marker":"[26]"},{"why":"A speaker-querying Whisper baseline against which DiCoW is compared.","marker":"[27]"},{"why":"Prior work defining the suppressive initialization used to insert FDDT transforms without breaking Whisper.","marker":"[35]"}],"fun_headline_variants":["Diarization tells Whisper who to listen to","Meeting transcription without speaker enrollment","Frame-level speaker masks guide Whisper's focus","Whisper's target speaker: decided by diarization","No speaker embeddings needed: DiCoW uses masks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model is trained only on perfect, hard diarization labels, so everything depends on the diarizer at test time producing labels close enough to the oracle ones; when it misses or confuses speech, DiCoW has no learned way to recover.","fun_headline_variants_meta":{"raw":{"variants":["Diarization tells Whisper who to listen to","Meeting transcription without speaker enrollment","Frame-level speaker masks guide Whisper's focus","Whisper's target speaker: decided by diarization","No speaker embeddings needed: DiCoW uses masks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000356,"raw_usage":{"total_tokens":2009,"prompt_tokens":1099,"completion_tokens":910,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":838}},"tokens_in":715,"tokens_out":910,"duration_ms":9526,"temperature":1.0,"reasoning_tokens":838,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:59:27.636827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released code, run inference on the reported test sets while randomly corrupting oracle diarization labels at rates from 5% to 30% (missed speech and speaker confusions), and check whether tcpWER rises in proportion to the corruption; the paper's assumption predicts a sharp rise with no recovery of missed segments.","supporting_citations":[{"cited_title":"Mccowan, J","cited_arxiv_id":null,"evidence_quote":"Provides the AMI meeting corpus used for training and evaluation."},{"cited_title":"Cosentino, M","cited_arxiv_id":null,"evidence_quote":"Provides the Libri2Mix synthetic mixture benchmark for comparison with prior methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LibriCSS overlapping-speech benchmark used for cross-dataset evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A prior target-speaker Whisper method that conditions on a speaker embedding, the approach DiCoW aims to replace."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A Whisper-based multi-talker and target-talker baseline using activation speech rather than embeddings."}],"review_version":1}