Pith. sign in

REVIEW 4 major objections 4 minor 46 references

MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-microphone Whisper rejects side talk and cuts word error rate by 4.95% on smart glasses.

desk verdict Interesting engineering with a coherent architecture, but the headline WERs are chosen on the development set, so the central claim is not yet established. read the letter →

arxiv 2507.05609 v1 pith:L2P6ERKU submitted 2025-07-08 eess.AS

Yang Liu , Li Wan , Yiteng Huang , Yong Xu , yangyang shi , Saurabh Adya , ming sun , Florian Metze This is my paper
classification eess.AS
keywords smartglassesside-talkrejectionmulti-microphonespeechprocessingWhisperMambagrouprelativepolicyoptimizationtarget-speakerASRworderrorrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Smart glasses capture wearer speech through open-field microphones, so bystander speech easily leaks into the transcript. This paper proposes MMW, a side-talk rejection multi-microphone Whisper system that fuses all microphone channels at the raw waveform level with a Tri-Mamba-based Mix Block, suppresses non-target frames through a Frame Diarization Mamba layer, and optimizes the whole pipeline with a multi-scale GRPO objective. The central claim is that MMW reduces word error rate by 4.95% in noisy conditions, from 67.81% WER for unmodified Whisper down to 3.71% with the full ablation configuration. If correct, the approach would make wearer-only transcription on smart glasses practical without explicit speaker embeddings or diarization annotations.

What carries the argument

The Mix Block is a Tri-Mamba-based frontend that fuses multichannel waveforms into embeddings while retaining bidirectional temporal context and inter-microphone phase. The Frame Diarization Mamba (FDM) layer sits between Mix Block output and Whisper's transformer blocks, applying class-conditioned transformations with frame probabilities $p_T$, $p_N$, and $p_O$ for target-only, non-target-only, and overlap. The multi-scale GRPO objective samples $G$ enhanced outputs and scores each with a composite reward combining frame-level normalized Hamming distance, utterance-level dominant-class match, and WER, then updates the enhancement policy via normalized advantages.

What would settle it

Evaluate the full MMW pipeline on unseen acoustic environments recorded with the same glasses but different head geometries; a WER closer to the FDM-only baseline (32.61%) than to 3.71% would indicate the Mix Block overfits the lab setup.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a Whisper encoder can be made wearer-aware by preserving inter-microphone phase at the waveform level rather than collapsing to a single log-mel channel. The proposed MMW pipeline feeds multi-channel waveforms through a Mix Block, inserts a Frame Diarization Mamba layer that masks frame features according to predicted target-only, non-target-only, and overlap classes, and trains with a composite GRPO reward combining frame-level Hamming distance, utterance-level dominant-class match, and WER. Ablations in the paper report WER falling from 67.81% (Whisper direct) to 32.61% with the FDM layer, 13.10% with frame and utterance rewards plus WER, and 3.71% once the Mix Block replaces beamforming plus log-mel frontends. Against baselines, MMW Whisper reaches 12.7% without ground-truth segmentation and 8.3% with it, and MMW plus a fine-tuned Llama 4 reaches 4.53% WER on the comparison set.

Load-bearing premise

The GRPO training objective requires ground-truth frame-level speaker labels for every frame, even though the paper claims to avoid explicit diarization labels.

Editorial extensions

If this is right

  • Wearable ASR can approach close-talk quality without requiring explicit speaker enrollment or diarization labels at inference time.
  • Streaming-compatible Mamba fusion allows real-time side-talk rejection inside Whisper-based assistants.
  • The Mix Block can be reused across hardware configurations, since only the spatial frontend needs retraining for a new microphone layout.
  • Fine-tuning an LLM on multi-channel spatial features (MMW + Llama 4) yields lower WER than fine-tuning the LLM on single-channel audio.
  • Preserving phase information at the waveform level matters more than classical beamforming and log-mel features for overlapping speech.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains rely on a proprietary dataset recorded with a fixed head shape (binaural dummy head) and Ray-Ban Meta glasses; real wearer variance in ear geometry, head movement, and acoustic environment may compress the 3.71% WER figure.
  • If the frame-level labels required by Eq. (8) are noisy or unavailable, the GRPO reward could be approximated by utterance-level and WER terms alone; the ablation suggests a large drop (13.10% without frame and utterance rewards), so label-free variants may need a proxy for $r_{frame}$.
  • The discrepancy between the abstract's 4.95% and Table 2's 4.53% for MMW + Llama 4 should be resolved before treating side-talk rejection as fully benchmarked.
  • A natural testable extension is to evaluate MMW on public spatial audio datasets (e.g., STARSS23 or Libri2Mix with real arrays) to confirm the Mix Block generalizes beyond the in-house setup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes MMW, a multi-microphone extension of Whisper for side-talk rejection on smart glasses. The system combines a Tri-Mamba Mix Block for raw-waveform multi-channel fusion, a Frame Diarization Mamba layer inserted into the Whisper encoder, and a GRPO-style training objective with frame-level, utterance-level, and WER rewards. Experiments on a newly recorded 5k-hour controlled playback dataset report a large WER reduction (to 3.71% in the ablation and 4.53–4.95% with Llama 4), and comparisons on Libri2Mix show gains over several target-speaker ASR baselines. The central claim is that MMW substantially improves wearer-only transcription in multi-talker conditions.

Significance. The proposed architecture is interesting and the controlled real-world dataset is potentially valuable; the Mix Block is a plausible way to preserve spatial phase information, and the use of GRPO to combine frame-level and utterance-level rewards with WER is a novel training formulation. If the reported results held on a truly held-out test set, the contribution would be significant for on-device ASR. However, the manuscript does not make code or data available, and the evaluation protocol as described prevents the headline results from being accepted as evidence of generalization. The paper also contains internal inconsistencies in the reported numbers and in the label requirements of the GRPO objective.

major comments (4)
  1. [§4.2] Section 4.2 states that 'Final evaluation metrics were based on the development set, selecting the checkpoint that achieved the lowest word error rate (WER).' All results in Section 5, the abstract, and the conclusion are then reported as final system performance. If the development set used for checkpoint selection is the same 22-hour evaluation set described in Section 4.1, every reported WER for the custom dataset is an in-sample number obtained by selecting the best checkpoint on the evaluation data. The paper's central claim—that MMW reduces WER in noisy conditions—is a generalization claim, so it requires a held-out test set that did not influence checkpoint selection, early stopping, or hyperparameters. As written, the reported gains do not establish held-out performance.
  2. [§3.3 / Eq. (8)] Section 3 claims the system works 'without relying on explicit diarization labels or speaker embeddings,' yet Eq. (8) defines r_frame using ground-truth frame labels yframe_1:T, and Eq. (12) includes this term in the overall reward. The training procedure as written requires frame-level speaker labels (target-only, non-target-only, overlap), which contradicts the label-free claim. The authors should state how these labels are obtained (for example, from the controlled playback schedule) and clarify what 'without explicit diarization labels' means; otherwise the GRPO objective cannot be implemented as described.
  3. [Abstract / Conclusion / Table 2] The abstract and conclusion report a 4.95% WER reduction for MMW with Llama 4, but Table 2 reports MMW + Llama 4 at 4.53%, and the conclusion's '3.71% in ablation studies and 4.95% when combined with Llama 4' does not match Table 1's 3.71% or Table 2's 4.53%. The headline number is inconsistent and must be reconciled before the results can be trusted.
  4. [§5.1 / Table 2] Table 2 reports comparisons 'on the Libri2Mix test-both set,' while Section 4.1 describes a new custom smart-glasses dataset; it is unclear whether Table 1 and Table 2 are evaluated on the same data or on different datasets, and no error bars or evaluation-set sizes are given. The comparison to baselines is therefore difficult to interpret, and the claim that MMW 'substantially outperforms' prior systems is not supported without a clear statement of which numbers come from which corpus and how variability was assessed.
minor comments (4)
  1. [§3.3] The term 'Multi-Scale' in 'Multi-Scale Group Relative Policy Optimization' is not defined; Eqs. (7)–(15) describe a single frame-level reward, a single utterance-level reward, and a WER reward, with no explicit multiscale decomposition. Please clarify the name or the objective.
  2. [§4.1 / §4.2] Section 4.1 reports a 22-hour evaluation set but no development set, while Section 4.2 refers to a development set for checkpoint selection; please specify the train/dev/test split and ensure the split names are consistent across the paper.
  3. [Throughout] There are several typographical errors, including 'Spacial information' in Section 3 and 'Y ouTube' in the reference list; a careful proofread is needed.
  4. [Tables 1 and 2] Table 1 is labeled 'evaluation set' while Table 2 is labeled 'Libri2Mix test-both set'; the manuscript should state explicitly which corpus and which split each table reports.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline WER reduction is checkpoint-selected on the same development set used for evaluation, so the main reported gain is partly an in-sample selection artifact; the architecture itself is not derived from its outputs.

  1. fitted input called prediction [Section 4.1-4.2 (dataset and training details), Section 5 / Table 1 caption]
    "Finally, we collect 5k hours training set and 22 hours evaluation set. ... Final evaluation metrics were based on the development set, selecting the checkpoint that achieved the lowest word error rate (WER). ... Table 1: Ablation study results on evaluation set."

    No separate test set is introduced after the 22-hour evaluation set of Section 4.1. Section 4.2 selects the checkpoint with the lowest WER on the development set, and Section 5 presents those same WERs as evaluation set results, including the 3.71% headline used in the abstract and conclusion. By construction, each reported number is the minimum over checkpoints of the WER on the data being reported, so it is a fitted selection statistic rather than a held-out prediction. The external Libri2Mix test-both results in Table 2 are independent evidence, making the circularity partial rather than total.

full rationale

The proposed components are specified by equations rather than assumed from the conclusion: Mix Block fusion (Eqs. 3-5), FDM masking (Eq. 6), and GRPO rewards (Eqs. 7-15) are concrete and are tested by ablation. Including WER in the GRPO reward (Eq. 11) is ordinary metric-based training, not itself circular, and would be unobjectionable with a held-out test set. The main circularity is that the development set doubles as the evaluation set for checkpoint selection, so the abstract and conclusion headline WER numbers are in-sample minima. The contradiction that Eq. (8) requires ground-truth frame labels while Section 3 claims no reliance on explicit diarization labels is an internal consistency issue, not a circular derivation. The Tri-Mamba self-citation [24] is an architectural borrowing, not a load-bearing uniqueness or theorem citation. Because Table 2's Libri2Mix test-both comparisons and the component ablations retain independent content, the paper is partially circular rather than fully reducible.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The framework rests on several domain assumptions and on undisclosed hyperparameters that are chosen by hand.

free parameters (4)
  • lambda 1, lambda 2, lambda 3 = not reported
    Weights in the composite GRPO reward (Eq. 12); values are not disclosed.
  • beta = not reported
    Exponential weighting factor in Eq. (7); not disclosed.
  • G = not reported
    Number of sampled outputs per input in GRPO (Eq. 13); not disclosed.
  • Mix Block and FDM hidden dimensions = not reported
    Architecture sizes are not specified for either proposed module.
assumptions (4)
  • domain assumption Multi-channel raw waveforms contain speaker-separating spatial cues without explicit microphone geometry
    Invoked in Section 3.1 to justify waveform-level fusion; no geometry-aware baseline is compared.
  • domain assumption Ground-truth frame-level speaker labels (T/N/O) are available for training
    Required by r_frame in Eq. (8); contradicts the Section 3 claim that MMW avoids explicit diarization labels.
  • domain assumption The lab playback dataset generalizes to real smart-glasses use
    Section 4.1 records a dummy head in an acoustic lab; no field recordings are included.
  • standard math Mamba/SSM theory and the Tri-Mamba block from [24] are valid building blocks
    Used to define the Mix Block and FDM layer, citing prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses." pith.science (2026). https://pith.science/paper/L2P6ERKU

@misc{pith2026250705609,
  author       = {Pith},
  title        = {Pith review of: MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2P6ERKU}},
  note         = {Machine review of arXiv:2507.05609}
}
read the original abstract

Smart glasses are increasingly positioned as the next-generation interface for ubiquitous access to large language models (LLMs). Nevertheless, achieving reliable interaction in real-world noisy environments remains a major challenge, particularly due to interference from side speech. In this work, we introduce a novel side-talk rejection multi-microphone Whisper (MMW) framework for smart glasses, incorporating three key innovations. First, we propose a Mix Block based on a Tri-Mamba architecture to effectively fuse multi-channel audio at the raw waveform level, while maintaining compatibility with streaming processing. Second, we design a Frame Diarization Mamba Layer to enhance frame-level side-talk suppression, facilitating more efficient fine-tuning of Whisper models. Third, we employ a Multi-Scale Group Relative Policy Optimization (GRPO) strategy to jointly optimize frame-level and utterance-level side speech suppression. Experimental evaluations demonstrate that the proposed MMW system can reduce the word error rate (WER) by 4.95\% in noisy conditions.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 35 canonical work pages

  1. [1]

    Their integratio n into wearable devices, such as smart glasses, has opened new opportunities for ubiquitous, voice-based interaction

    INTRODUCTION Large language models have become central to a wide range of AI applications, including dialogue systems, content ge n- eration, and multimodal reasoning [1, 2]. Their integratio n into wearable devices, such as smart glasses, has opened new opportunities for ubiquitous, voice-based interaction. T hese devices promise to serve as natural lang...

    work page

  2. [2]

    RELA TED WORK 2.1. Whisper OpenAI’s Whisper is a powerful end-to-end automatic speech recognition (ASR) system based on a Transformer encoder- decoder architecture, trained through large-scale weak su - pervision on multilingual audio datasets [19]. Whisper is available in multiple model sizes, including tiny (39M pa- rameters), base (74M parameters), sma...

    work page

  3. [3]

    The pro- posed model leverages multi-channel audio waveforms to guide the transcription process, as illustrated in Fig

    SIDE-TALK REJECTION MUL TI-MICROPHONE WHISPER This section introduces side-talk rejection multi-microp hone whisper (MMW), an extension of the Whisper architecture tailored for wearer-centric ASR on smart glasses. The pro- posed model leverages multi-channel audio waveforms to guide the transcription process, as illustrated in Fig. 1. To enable wearer-awa...

    work page

  4. [4]

    dataset Several spatial audio datasets have been introduced to support model development, including Y ouTube-360 [32], Y ouTube- ASMR [33], Pano-A VQA [34], and STARSS23 [35]

    EXPERIMENTS 4.1. dataset Several spatial audio datasets have been introduced to support model development, including Y ouTube-360 [32], Y ouTube- ASMR [33], Pano-A VQA [34], and STARSS23 [35]. How- ever, many of these datasets suffer from inconsistent quali ty and lack critical annotations, such as sound source directi on or distance, limiting their appli...

    work page

  5. [5]

    We first directly evaluate the original Whisper model on our multi- microphone dataset, achieving a Word Error Rate of 67.81%

    ABLA TION STUDY We conduct an ablation study to evaluate the contribution of each proposed component, as shown in Table 1. We first directly evaluate the original Whisper model on our multi- microphone dataset, achieving a Word Error Rate of 67.81%. The high error rate indicates that the unmodified Whisper model transcribes both wearer and bystander speech ...

    work page

  6. [6]

    CONCLUSION We proposed Side-Talk Rejection Multi-Microphone Whis- per (MMW), a new framework that enhances wearer-centric ASR on smart glasses by integrating a Mix Block for multi- channel raw waveform fusion, a Frame Diarization Mamba Layer for side-talk suppression, and a Multi-Scale Group Relative Policy Optimization (GRPO) strategy. Experiment s on a ...

    work page

  7. [7]

    Sound event localization and detection of overlapping sources using convolutional re- current neural networks,

    Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen, “Sound event localization and detection of overlapping sources using convolutional re- current neural networks,” IEEE Journal of Selected T op- ics in Signal Processing, vol. 13, no. 1, pp. 34–48, 2018

    work page 2018

  8. [8]

    Gpt-4 technical report,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

Show all 46 references
  1. [9]

    For example, DOAnet [7] leverages a hybrid CNN-RNN architecture to estimate DoAs for multiple sound sources under challenging acoustic en- vironments

    have shown promise in learning spatial features directl y from waveforms or spectrograms. For example, DOAnet [7] leverages a hybrid CNN-RNN architecture to estimate DoAs for multiple sound sources under challenging acoustic en- vironments. Nonetheless, most of these models ar...

    2025 arXiv

  2. [10]

    Llama: Open and efficient foundation lan- guage models,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al., “Llama: Open and efficient foundation lan- guage models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  3. [11]

    Beamform- ing: A versatile approach to spatial filtering,

    Barry D V an V een and Kevin M Buckley, “Beamform- ing: A versatile approach to spatial filtering,” IEEE assp magazine, vol. 5, no. 2, pp. 4–24, 1988

    1988

  4. [12]

    The generalized correlation method for estimation of time delay,

    Charles Knapp and Glifford Carter, “The generalized correlation method for estimation of time delay,” IEEE transactions on acoustics, speech, and signal process- ing, vol. 24, no. 4, pp. 320–327, 2003

    2003

  5. [13]

    Multiple emitter location and signal parameter estimation,

    Ralph Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE transactions on antennas and propagation, vol. 34, no. 3, pp. 276–280, 1986

    1986

  6. [14]

    Esprit-estimation of signal parameters via rotational invariance techniques,

    Richard Roy and Thomas Kailath, “Esprit-estimation of signal parameters via rotational invariance techniques,” IEEE Transactions on acoustics, speech, and signal pro- cessing, vol. 37, no. 7, pp. 984–995, 1989

    1989

  7. [15]

    Convolutional gated recurrent neural network incorporating spatial features for audio tagging,

    Y ong Xu, Qiuqiang Kong, Qiang Huang, Wenwu Wang, and Mark D Plumbley, “Convolutional gated recurrent neural network incorporating spatial features for audio tagging,” in 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, 2017, pp. 3461–3466

    2017

  8. [16]

    Many-to-many audio spectrogram tansformer: Trans- former for sound event localization and detection.,

    Sooyoung Park, Y oungho Jeong, and Taejin Lee, “Many-to-many audio spectrogram tansformer: Trans- former for sound event localization and detection.,” in DCASE, 2021, pp. 105–109

    2021

  9. [17]

    Bat: Learning to reason about spatial sounds with large language mod- els,

    Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath, “Bat: Learning to reason about spatial sounds with large language mod- els,” arXiv preprint arXiv:2402.01591 , 2024

    2024 arXiv

  10. [18]

    Can large language models under- stand spatial audio?,

    Changli Tang, Wenyi Y u, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Jun Zhang, Lu Lu, Zejun Ma, Y uxuan Wang, et al., “Can large language models under- stand spatial audio?,” arXiv preprint arXiv:2406.07914 , 2024

    2024 arXiv

  11. [19]

    Soundspaces 2.0: A simulation platform for visual-acoustic learn- ing,

    Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman, “Soundspaces 2.0: A simulation platform for visual-acoustic learn- ing,” Advances in Neural Information Processing Sys- tems, vol. 35,...

    2022

  12. [20]

    Srp-dnn: Learn- ing direct-path phase difference for multiple moving sound source localization,

    Bing Y ang, Hong Liu, and Xiaofei Li, “Srp-dnn: Learn- ing direct-path phase difference for multiple moving sound source localization,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP). IEEE, 2022, pp. 721–725

    2022

  13. [21]

    Joint learning of audio– visual saliency prediction and sound source localization on multi-face videos,

    Minglang Qiao, Y ufan Liu, Mai Xu, Xin Deng, Bing Li, Weiming Hu, and Ali Borji, “Joint learning of audio– visual saliency prediction and sound source localization on multi-face videos,” International Journal of Com- puter Vision, vol. 132, no. 6, pp. 2003–2025, 2024

    2003

  14. [22]

    Learning spatially-aware language and audio embed- dings,

    Bhavika Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso, Elena Menyaylenko, Barry-John Theobald, Jonathan Sheaffer, and Miguel Sarabia, “Learning spatially-aware language and audio embed- dings,” Advances in Neural Information Processing Sys- tems, vol. 37, pp. 33505–33...

    2024

  15. [23]

    Salmonn: Towards generic hearing abilities for large language models,

    Changli Tang, Wenyi Y u, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang, “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023

    2023 arXiv

  16. [24]

    Gama: A large audio-language model with advanced audio un- derstanding and complex reasoning abilities,

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Ki- ran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha, “Gama: A large audio-language model with advanced audio un- derstanding and complex reasoning abilities,” arXiv preprint arXiv:2406...

    2024 arXiv

  17. [25]

    Lora: Low-rank adaptation of large lan- guage models.,

    Edward J Hu, Y elong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Y uanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al., “Lora: Low-rank adaptation of large lan- guage models.,” ICLR, vol. 1, no. 2, pp. 3, 2022

    2022

  18. [26]

    Robust speech recognition via large-scale weak supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28492–28518

    2023

  19. [27]

    Extending whisper with prompt tuning to target-speaker asr,

    Hao Ma, Zhiyuan Peng, Mingjie Shao, Jing Li, and Ju Liu, “Extending whisper with prompt tuning to target-speaker asr,” in ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12516–12520

    2024

  20. [28]

    Em- powering whisper as a joint multi-talker and target- talker speech recognition system,

    Lingwei Meng, Jiawen Kang, Y uejiao Wang, Zengrui Jin, Xixin Wu, Xunying Liu, and Helen Meng, “Em- powering whisper as a joint multi-talker and target- talker speech recognition system,” arXiv preprint arXiv:2407.09817, 2024

    2024 arXiv

  21. [29]

    Dicow: Diarization-conditioned whisper for target speaker automatic speech recognition,

    Alexander Polok, Dominik Klement, Martin Kocour, Jiangyu Han, Federico Landini, Bolaji Y usuf, Matthew Wiesner, Sanjeev Khudanpur, Jan ˇCernock` y, and Luk´ aˇ s Burget, “Dicow: Diarization-conditioned whisper for target speaker automatic speech recognition,” arXiv preprint ar...

    2024 arXiv

  22. [30]

    Target speaker asr with whisper,

    Alexander Polok, Dominik Klement, Matthew Wies- ner, Sanjeev Khudanpur, Jan ˇCernock` y, and Luk´ aˇ s Bur- get, “Target speaker asr with whisper,” in ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2025, pp. 1–5

    2025

  23. [31]

    Masv: Speaker verification with global and local context mamba,

    Y ang Liu, Li Wan, Yiteng Huang, Ming Sun, Y angyang Shi, and Florian Metze, “Masv: Speaker verification with global and local context mamba,” arXiv preprint arXiv:2412.10989, 2024

    2024 arXiv

  24. [32]

    Exploration of whisper fine-tuning strategies for low-resource asr,

    Y unpeng Liu, Xukui Y ang, and Dan Qu, “Exploration of whisper fine-tuning strategies for low-resource asr,” EURASIP Journal on Audio, Speech, and Music Pro- cessing, vol. 2024, no. 1, pp. 29, 2024

    2024

  25. [33]

    Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recogni- tion and translation,

    Andrew Rouditchenko, Y uan Gong, Samuel Thomas, Leonid Karlinsky, Hilde Kuehne, Rogerio Feris, and James Glass, “Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recogni- tion and translation,” arXiv preprint arXiv:2406.10082 , 2024

    2024 arXiv

  26. [34]

    Efficientl y modeling long sequences with structured state spaces,

    Albert Gu, Karan Goel, and Christopher R´ e, “Efficientl y modeling long sequences with structured state spaces,” arXiv preprint arXiv:2111.00396 , 2021

    2021 arXiv

  27. [35]

    Hippo: Recurrent memory with op- timal polynomial projections,

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R´ e, “Hippo: Recurrent memory with op- timal polynomial projections,” Advances in neural in- formation processing systems , vol. 33, pp. 1474–1487, 2020

    2020

  28. [36]

    Attention is all you need,

    Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Ad- vances in neural information processing systems , vol. 30, 2017

    2017

  29. [37]

    Long range arena: A benchmark for efficient transformers,

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Y ang, Se- bastian Ruder, and Donald Metzler, “Long range arena: A benchmark for efficient transformers,” arXiv preprint arXiv:2011.04006, 2020

    2011 arXiv

  30. [38]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,

    Tri Dao and Albert Gu, “Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,” arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  31. [39]

    Learn- ing representations from audio-visual spatial align- ment,

    Pedro Morgado, Yi Li, and Nuno Nvasconcelos, “Learn- ing representations from audio-visual spatial align- ment,” Advances in Neural Information Processing Sys- tems, vol. 33, pp. 4733–4744, 2020

    2020

  32. [40]

    Telling left from right: Learning spatial correspon- dence of sight and sound,

    Karren Y ang, Bryan Russell, and Justin Salamon, “Telling left from right: Learning spatial correspon- dence of sight and sound,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9932–9941

    2020

  33. [41]

    Pano-avqa: Grounded audio- visual question answering on 360deg videos,

    Heeseung Y un, Y oungjae Y u, Wonsuk Y ang, Kangil Lee, and Gunhee Kim, “Pano-avqa: Grounded audio- visual question answering on 360deg videos,” in Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2031–2041

    2021

  34. [42]

    Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,

    Kazuki Shimada, Archontis Politis, Parthasaarathy Su - darsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Y uichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al., “Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotempo...

    2023

  35. [43]

    Decoupled weight decay regularization,

    Ilya Loshchilov and Frank Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  36. [44]

    Weakly-supervised speech pre-training: A case study on target speech recognition,

    Wangyou Zhang and Y anmin Qian, “Weakly-supervised speech pre-training: A case study on target speech recognition,” arXiv preprint arXiv:2305.16286 , 2023

    2023 arXiv

  37. [45]

    Surt 2.0: Advances in transducer-based multi-talker speech recognition,

    Desh Raj, Daniel Povey, and Sanjeev Khudanpur, “Surt 2.0: Advances in transducer-based multi-talker speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 3800– 3813, 2023

    2023

  38. [46]

    Notsofar-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,

    Alon Vinnikov, Amir Ivry, Aviv Hurvitz, Igor Abramovski, Sharon Koubi, Ilya Gurvich, Shai Peer, Xiong Xiao, Benjamin Martinez Elizalde, Naoyuki Kanda, et al., “Notsofar-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,” arXiv preprint arXiv:2401...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.