Pith. sign in

REVIEW 2 major objections 6 minor 79 references

Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model trained on audio and video can be deployed with audio only and still recover most of the audiovisual advantage in speech tasks, at a fraction of the compute.

desk verdict Solid engineering result with a credible AVSE evaluation and a plausible but under-supported mechanism claim; the missing distillation baseline is the key gap, not a reason to reject. read the letter →

arxiv 2501.18157 v1 pith:VFOTSZL5 submitted 2025-01-30 cs.SD cs.CVcs.MMeess.AS

Joanna Hong , Sanjeel Parekh , Honglie Chen , Jacob Donley , Ke Tan , Buye Xu , Anurag Kumar This is my paper · ORCID
classification cs.SDcs.CVcs.MMeess.AS
keywords audiovisualspeechprocessingenhancementmultimodaltrainingunimodaldeploymentcodebookfeatureestimationrecognitionactivespeakerdetectionmodelefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal models that combine audio and video improve speech tasks, but a camera feed is often unavailable, costly, or hard to keep synchronized at inference time. This paper proposes MUTUD, a training scheme that uses audio and video together during training yet needs only one modality when deployed. Its TAME module learns paired audio and video codebooks, then estimates the missing video features from the audio features during inference, so the video encoder can be discarded. On speech enhancement, recognition, and active speaker detection, the authors report that this recovers most of the gap between audio-only and full audiovisual models while cutting the deployed model's compute and parameters by roughly 80 percent. If the finding holds, practitioners could get much of the robustness benefit of audiovisual learning without requiring a camera or synchronized video at run time.

What carries the argument

The central mechanism is the Temporally Aligned Modality feature Estimation (TAME) module, which maintains separate codebooks for audio and video, each with K blocks of N learned codes, where K is the audio-to-video frame-rate ratio. Audio features at each temporal location are converted into a soft distribution over their codebook, and that same distribution is used to read a weighted combination of video codes, producing an estimated video feature that replaces the real video feature at inference. Three training losses do the work: a video self-reconstruction loss, an audio-to-video reconstruction loss, and a KL-divergence loss that makes each audio codebook's distribution match the corresponding video codebook's distribution, with the task loss on top. After training, the video encoder is discarded, leaving only the audio encoder, TAME, and the task decoder.

What would settle it

If disabling the learned codebooks, for example by replacing them with random fixed vectors while keeping the same parameters, leaves the gains over the matched-parameter audio-only model unchanged, then the codebook mechanism is not what carries the visual benefit and the central claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that multimodal training can be separated from unimodal deployment: a model trained on audiovisual pairs can be run at inference with audio alone, losing only a small part of the audiovisual advantage while costing about as much as the unimodal model. In the main speech-enhancement experiment at -15 dB SNR with three background noises, MUTUD reaches 54.0 STOI and -2.19 dB SI-SDR, compared with 51.5 STOI and -2.82 dB SI-SDR for the audio-only model and 62.7 STOI and -0.86 dB SI-SDR for the full audiovisual model. The effect is attributed to TAME's learned modality-specific codebooks, and the paper shows similar patterns on a smaller backbone, on the VisualVoice backbone, on audiovisual speech recognition, and on egocentric active speaker detection, as well as on the out-of-domain DNS Challenge evaluation set. The deployed enhancement model uses 3.635M parameters and 1.593G MACs, versus 15.736M parameters and 9.324G MACs for the audiovisual model.

Load-bearing premise

The argument rests on the assumption that the visual benefit comes from the learned codebook lookup tables, and that those tables keep retrieving useful video information at deployment-time SNRs and on audio-only datasets beyond the training mixture.

Editorial extensions

If this is right

  • An audio-only deployment recovers most of the audiovisual intelligibility gain: at -15 dB, STOI is 54.0 for MUTUD versus 51.5 for audio-only and 62.7 for audiovisual.
  • The deployed enhancement model runs at 1.593G MACs with 3.635M parameters, versus 9.324G MACs and 15.736M parameters for the audiovisual model, a drop of roughly 80 percent.
  • The benefit transfers across backbones and tasks: the authors report consistent gains on two GCRN-style backbones, the VisualVoice backbone, audiovisual speech recognition (word error rate 73.56% vs 93.32% for audio-only at -15 dB), and active speaker detection (mean average precision 86.5% vs 82.3% video-only and 87.6% audiovisual).
  • MUTUD also improves over its audio-only counterpart on the out-of-domain DNS Challenge evaluation set, showing the visual knowledge gained during training is usable when no video exists at inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated extension is that the same pairwise-codebook recipe could drop other modalities at inference, such as text, IMU, or microphone-array signals, since the paper's MUTUD formulation is written for arbitrary modality sets.
  • Because the reported savings are measured on the enhancement backbone, the practical deployment saving is likely larger once video capture, lip-region cropping, and synchronization overhead are removed from the pipeline.
  • The fixed audio-to-video frame-rate ratio K used by TAME suggests a testable extension: evaluating the trained model under frame jitter or mismatched frame rates would show whether the codebook alignment generalizes to unsynchronized real-world sensors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes MUTUD (Multimodal Training and Unimodal Deployment), a training strategy that uses audiovisual data during training but only audio (or only video) during inference. The key component is the Temporally Aligned Modality feature Estimation (TAME) module, which learns audio and video codebooks with K temporally aligned blocks and estimates video features from audio via a softmax-weighted combination of code vectors (Eq. 5). Three losses train the codebooks and the audio-to-video mapping: video self-reconstruction (Eq. 6), audio-to-video feature reconstruction (Eq. 7), and a KL divergence between codebook distributions (Eq. 8). The paper reports experiments on audiovisual speech enhancement (three backbones, two noise conditions, SNRs 5 to -15 dB, plus a DNS Challenge out-of-domain set), audiovisual speech recognition, and audiovisual active speaker detection. The headline result is that MUTUD outperforms audio-only and matched-parameter audio-only baselines, especially at low SNR, while its parameters and MACs are close to audio-only and about 77-83% lower than the full audiovisual AVSE model.

Significance. The practical question addressed is real, and the empirical scope is a genuine strength: three tasks, multiple backbones, matched-parameter controls for AVSE, codebook-size ablations, and an out-of-domain DNS evaluation. If the mechanism claim held, MUTUD would be a useful recipe for on-device speech processing that keeps most of the audiovisual gain without requiring a camera or video encoder at deployment. The headline AVSE numbers do show consistent gains over audio-only, and the efficiency numbers are concrete. However, the paper does not yet isolate the TAME codebook mechanism from the auxiliary training losses, and the AVSR support is much weaker at high SNR. These issues are addressable with additional control experiments and a more careful statement of the claims.

major comments (2)
  1. [Sec. V-A, Table I; Sec. III-B, Eqs. (6)-(8)] The matched-parameter audio-only control in Table I rules out raw parameter count as the source of the gains, but it does not isolate the TAME codebook mechanism from the training objectives in Eqs. (6)-(8). At inference the model is a deterministic function of audio alone, so any visual benefit must enter through training-time losses. To support the claim that the codebook retrieval in Eq. (5) is the load-bearing component, the authors should add a knowledge-distillation baseline that trains the same audio-only backbone to match the audiovisual teacher's enhanced output or hidden features (video available only during training), or an ablation that replaces the codebook estimator with an MLP/linear predictor of video features from audio features. Without such a control, the feature-similarity analysis in Fig. 3 is not independent evidence: the high cosine similarity between Fv and hat Fv is an explicit optimization target of Eq. (7), not a validation that the codebooks, rather than the auxiliary losses, carry the visual benefit.
  2. [Sec. V-F2, Table VIII] The AVSR conclusion is stronger than the data support. At 5 dB SNR, MUTUD reduces WER from 12.24 to 11.71 (about 0.5 points absolute, 4% relative), and at 0 dB the relative reduction is about 8.6%; the 'substantial reduction in WER' statement in Sec. V-F2 is really supported only at -5 dB and below. The paper should either report confidence intervals or statistical significance for these differences, or explicitly restrict the 'considerable extent' claim to low-SNR conditions for the ASR task.
minor comments (6)
  1. [Sec. III-B, Eq. (8)] The summation in Eq. (8) is written over k=1..K, while codebooks in Eqs. (1)-(4) are indexed 0..K-1; also the distributions P^k and Q^k should carry the time index t, or the sum over t should be explicit. Please make the notation consistent.
  2. [Sec. V-C, Table VI] The text says MUTUD is 'smaller almost by a factor of 5', but the ratio is 15.736M / 3.635M, which is about 4.33; the parameter reduction is 76.9%, while the MAC reduction is 82.9%. Please report the exact percentages and distinguish the two quantities.
  3. [Sec. V-C] Efficiency is reported only for the AVSE backbone. Because the abstract and introduction claim efficiency benefits for the framework generally, please add at least parameter/MAC counts for the AVSR and AV-ASD configurations, or state explicitly that efficiency was evaluated only for AVSE.
  4. [Fig. 3] The legend is not visible in the current rendering; please label the curves with the three feature-pair comparisons and state which color corresponds to cosine similarity and which to L2 distance.
  5. [Sec. V-F, Table VIII] Table VIII does not include an audio-only matched-parameter baseline (unlike Table I), so the ASR comparison does not rule out capacity differences; please either add such a baseline or note this limitation.
  6. [Throughout] There are several typos, e.g., 'defnitely' in the Introduction, 'multiplicted' in Sec. V-F1, 'V oxCeleb2' in Sec. V-G, and 'state-the-art' in Sec. V-B.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the central MUTUD claims rest on held-out enhancement benchmarks and are not derived from the fitted quantities; only the internal feature-similarity diagnostic is partly self-consistency.

  1. other [Section III-C, Eq. (7) and Section V-D (TAME Module Analysis, Fig. 3)]
    "La→v = Tv−1X t=0 K−1X k=0 ∥ ˆf K·t+k v − f t v∥2 2 (7) ... Figure 3 clearly indicates that the cosine similarity between the estimated video features and the original video features is high, around 0.94 ... The high similarity between the estimated and original video features, while having low similarity between the estimated video and audio features evidence that TAME is not just regurgitating audio features but is actually functioning as designed (use audio information to get video information)."

    The Fig. 3 validation measures agreement between the estimated video features and the original video features, which is precisely the quantity minimized by the training objective in Eq. (7). Moreover, the 'original' video features Fv come from a video encoder trained jointly with TAME and the task loss, so the diagnostic shows that the estimator matches its own learned training target rather than an externally grounded visual ground truth. This is an internal self-consistency issue and is not load-bearing for the main claim, because the downstream STOI, SISDR, PESQ, and DNS Challenge results are evaluated on held-out data and do not use Fv at inference.

full rationale

The paper's central claim is that MUTUD reduces the gap between audiovisual and audio-only models while cutting compute, and that claim is supported by held-out evaluations on LRS3 test mixtures and the out-of-domain DNS Challenge set. There is no equation in the paper that reduces a reported performance number to a fitted parameter, and no fitted input is renamed as a prediction: the TAME module is trained to estimate video features from audio, but the reported STOI/SISDR/PESQ gains are measured on test inputs and compared against independent audio-only and audiovisual baselines. The parameter-matched audio-only control addresses raw capacity, although it does not isolate the TAME-specific losses from a generic auxiliary training signal; that is a mechanism-attribution gap, not a circularity. Self-citations appear mainly as baseline architectures and related memory-bridging work, and none is used as an external uniqueness theorem or as the sole support for the central claim. The compute and size reductions in Table VI follow by direct counting of parameters and MACs, not by construction from the method's assumptions. The only notable circularity-adjacent element is the feature-similarity analysis in Fig. 3, which verifies that the estimator matches its own training target; this is a minor internal diagnostic and does not undermine the independent empirical comparisons. Overall, no load-bearing circular reduction is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities; its contributions are architectural and procedural. The central claim depends on three hand-set hyperparameters (loss weights, codebook size, temporal ratio K) and on the domain assumption that video encoder features are a sufficient target for audio-based retrieval. The matched-parameter control is the paper's main defense against the alternative explanation that gains come from extra capacity.

free parameters (4)
  • Loss weights alpha, beta, gamma, lambda = 1.0, 1.0, 1.0, 0.01
    Set by hand in Sec IV-B; no sensitivity study. They control the balance between feature reconstruction, cross-modal codebook alignment, and task loss; if the task loss weight lambda is too low, the TAME could drift away from task-relevant features.
  • Codebook size N (number of codes per codebook) = 32
    Chosen after ablation over {8,16,32,64} in Sec V-E; Table VII shows performance increases with N, saturating at 32. This is a capacity hyperparameter fitted to the validation or test performance, not derived from theory.
  • Temporal ratio K = 4 for AVSE
    Assumed equal to the audio/video frame-rate ratio (audio 100 Hz, video 25 Hz, Sec III-B). For AVSR and AV-ASD, video features are upsampled to audio rate, so the K relation is not clearly defined; this is a fixed design assumption rather than a fitted value, but it shapes the architecture.
  • Softmax temperature tau = not reported
    Appears in Eqs 3-4 and controls the sharpness of codebook selection; its value is not given, so the codebook retrieval behavior cannot be fully replicated.
assumptions (4)
  • domain assumption A fixed temporal alignment exists between audio and video features, with one video frame corresponding to K audio frames.
    The TAME design in Sec III-B builds codebooks in K temporal blocks to exploit this; the AVSR/AV-ASD adaptations upsample video features, which may violate the assumption.
  • domain assumption Video encoder features (ResNet-18 or 3D conv) are a sufficient target for cross-modal estimation; recovering them from audio recovers the task-relevant visual information.
    The La->v loss (Eq 7) and the task loss using both real and estimated video features assume that the visual signal useful for enhancement or recognition is fully contained in the video encoder's deep features.
  • domain assumption The training noise distribution (DNS noise set, SNR -15 to 10 dB) is representative of the test and deployment conditions, including the out-of-domain DNS eval set.
    Table V supports this for enhancement, but AVSR and AV-ASD results are only reported on the training noise conditions.
  • standard math Softmax, cosine similarity, and KL divergence are used as standard operations (Eqs 1-8) without further proof.
    These are standard operations; no new mathematical claims are made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment." pith.science (2026). https://pith.science/paper/VFOTSZL5

@misc{pith2026250118157,
  author       = {Pith},
  title        = {Pith review of: Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VFOTSZL5}},
  note         = {Machine review of arXiv:2501.18157}
}
read the original abstract

Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions frequently lead to improvements in performance and may even be critical in certain cases, they come with several constraints such as increased sensory requirements, computational cost, and modality synchronization, to mention a few. These challenges constrain the direct uses of these multimodal solutions in real-world applications. In this work, we develop approaches where the learning happens with all available modalities but the deployment or inference is done with just one or reduced modalities. To do so, we propose a Multimodal Training and Unimodal Deployment (MUTUD) framework which includes a Temporally Aligned Modality feature Estimation (TAME) module that can estimate information from missing modality using modalities present during inference. This innovative approach facilitates the integration of information across different modalities, enhancing the overall inference process by leveraging the strengths of each modality to compensate for the absence of certain modalities during inference. We apply MUTUD to various audiovisual speech tasks and show that it can reduce the performance gap between the multimodal and corresponding unimodal models to a considerable extent. MUTUD can achieve this while reducing the model size and compute compared to multimodal models, in some cases by almost 80%.

Figures

Figures reproduced from arXiv: 2501.18157 by the authors.

Figure 1
Figure 1. (a) The left panel shows a comparison between conventional audiovisual speech processing and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. MUTUD bridges the gap between Audiovi￾sual and Audio-only models. Performance (in %) of different methods relative to the gain Audiovisual brings in average intelligibility (STOI) of noisy speech samples. MUTUD is able to recover most of performance gains of the Audiovisual model across different SNRs. For example, at -5dB SNR Audio￾only is at 86.0% of Audioivisual model whereas MUTUD is at 93.4% of Audiovisual mode… view at source ↗
Figure 6
Figure 6. Figure 6 shows the variation in the usage [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: Cosine similarity (red) and ℓ2 distance (blue) between video features and estimated video features, video and audio features, and estimated video and audio features for different SNRs. snr 0 snr -5 snr -10 snr -15 F!!"# F!"# F$%# snr 5 F!!"# F!"# F$%# F!!"# F!"# F$%# F…
Figure 4
Figure 4. Figure 4: TSNE visualization of the estimated video features [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the learned audio and video codebooks ( [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the probability distribution [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 59 canonical work pages

  1. [1]

    La-voce: Low-snr audio-visual speech enhancement using neural vocoders,

    R. Mira, B. Xu, J. Donley, A. Kumar, S. Petridis, V . K. Ithapu, and M. Pantic, “La-voce: Low-snr audio-visual speech enhancement using neural vocoders,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

    work page 2023

  2. [2]

    Discriminative multi-modality speech recognition,

    B. Xu, C. Lu, Y . Guo, and J. Wang, “Discriminative multi-modality speech recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14 433–14 442

    work page 2020

  3. [3]

    End-to-end audio-visual speech recognition with conformers,

    P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 7613–7617. 15

    work page 2021

  4. [4]

    Visual context-driven audio feature enhancement for robust end- to-end audio-visual speech recognition,

    J. Hong, M. Kim, D. Yoo, and Y . M. Ro, “Visual context-driven audio feature enhancement for robust end- to-end audio-visual speech recognition,” arXiv preprint arXiv:2207.06020, 2022

    arXiv 2022

  5. [5]

    Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring,

    J. Hong, M. Kim, J. Choi, and Y . M. Ro, “Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 783–18 794

    work page 2023

  6. [6]

    Seeing to hear better: evidence for early audio-visual interactions in speech identification,

    J.-L. Schwartz, F. Berthommier, and C. Savariaux, “Seeing to hear better: evidence for early audio-visual interactions in speech identification,” Cognition, vol. 93, no. 2, pp. B69– B78, 2004

    work page 2004

  7. [7]

    Burnham, R

    D. Burnham, R. Campbell, G. Away, and B. Dodd, Hearing eye II: the psychology of speechreading and auditory-visual speech. Psychology Press, 2013

    work page 2013

  8. [8]

    Speech enhancement with lstm recurrent neural networks and its application to noise-robust asr,

    F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, “Speech enhancement with lstm recurrent neural networks and its application to noise-robust asr,” in Latent Variable Analysis and Signal Separation: 12th International Conference, LVA/ICA 2015, Liberec, Czech Republic, August 25-28, 2015, Proceedings 12 . Springe...

    work page 2015

Show all 79 references
  1. [9]

    Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement,

    K. Tan and D. Wang, “Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 28, pp. 380–390, 2019

    2019

  2. [10]

    Complex spectral mapping for single-and multi-channel speech enhancement and robust asr,

    Z.-Q. Wang, P. Wang, and D. Wang, “Complex spectral mapping for single-and multi-channel speech enhancement and robust asr,” IEEE/ACM transactions on audio, speech, and language processing , vol. 28, pp. 1778–1787, 2020

    2020

  3. [11]

    Towards efficient models for real-time deep noise suppression,

    S. Braun, H. Gamper, C. K. Reddy, and I. Tashev, “Towards efficient models for real-time deep noise suppression,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 656–660

    2021

  4. [12]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltruˇsaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE transac- tions on pattern analysis and machine intelligence , vol. 41, no. 2, pp. 423–443, 2018

    2018

  5. [13]

    A theory of multimodal learning,

    Z. Lu, “A theory of multimodal learning,” Advances in Neural Information Processing Systems, vol. 36, pp. 57 244– 57 255, 2023

    2023

  6. [14]

    Visual speech enhancement,

    A. Gabbay, A. Shamir, and S. Peleg, “Visual speech enhancement,” arXiv preprint arXiv:1711.08789 , 2017

    2017 arXiv

  7. [15]

    The con- versation: Deep audio-visual speech enhancement,

    T. Afouras, J. S. Chung, and A. Zisserman, “The con- versation: Deep audio-visual speech enhancement,” arXiv preprint arXiv:1804.04121, 2018

    2018 arXiv

  8. [16]

    Visualvoice: Audio-visual speech separation with cross-modal consistency,

    R. Gao and K. Grauman, “Visualvoice: Audio-visual speech separation with cross-modal consistency,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021, pp. 15 490–15 500

    2021

  9. [17]

    Audio-visual speech codecs: Rethinking audio- visual speech enhancement by re-synthesis,

    K. Yang, D. Markovi ´c, S. Krenn, V . Agrawal, and A. Richard, “Audio-visual speech codecs: Rethinking audio- visual speech enhancement by re-synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8227–8237

    2022

  10. [18]

    Audio-visual scene analysis with self-supervised multisensory features,

    A. Owens and A. A. Efros, “Audio-visual scene analysis with self-supervised multisensory features,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 631–648

    2018

  11. [19]

    Audio-visual speech enhancement using multimodal deep convolutional neural networks,

    J.-C. Hou, S.-S. Wang, Y .-H. Lai, Y . Tsao, H.-W. Chang, and H.-M. Wang, “Audio-visual speech enhancement using multimodal deep convolutional neural networks,” IEEE Transactions on Emerging Topics in Computational Intelli- gence, vol. 2, no. 2, pp. 117–128, 2018

    2018

  12. [20]

    Audio-visual deep learning for noise robust speech recognition,

    J. Huang and B. Kingsbury, “Audio-visual deep learning for noise robust speech recognition,” in2013 IEEE international conference on acoustics, speech and signal processing . IEEE, 2013, pp. 7596–7599

    2013

  13. [21]

    Deep multimodal learning for audio-visual speech recognition,

    Y . Mroueh, E. Marcheret, and V . Goel, “Deep multimodal learning for audio-visual speech recognition,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 2130–2134

    2015

  14. [22]

    Audio-visual speech recognition using deep learning,

    K. Noda, Y . Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata, “Audio-visual speech recognition using deep learning,” Applied Intelligence, vol. 42, no. 4, pp. 722–737, 2015

    2015

  15. [23]

    Robust audio-visual speech recognition under noisy audio-video conditions,

    D. Stewart, R. Seymour, A. Pass, and J. Ming, “Robust audio-visual speech recognition under noisy audio-video conditions,” IEEE transactions on cybernetics, vol. 44, no. 2, pp. 175–184, 2013

    2013

  16. [24]

    Audio-visual speaker detection using dynamic bayesian networks,

    A. Garg, V . Pavlovic, and J. M. Rehg, “Audio-visual speaker detection using dynamic bayesian networks,” in Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) . IEEE, 2000, pp. 384–390

    2000

  17. [25]

    Look who’s talking: Speaker detection using video and audio correlation,

    R. Cutler and L. Davis, “Look who’s talking: Speaker detection using video and audio correlation,” in 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No. 00TH8532), vol. 3. IEEE, 2...

    2000

  18. [26]

    Active speaker detection with audio-visual co-training,

    P. Chakravarty, J. Zegers, T. Tuytelaars, and H. Van hamme, “Active speaker detection with audio-visual co-training,” in Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016, pp. 312–316

    2016

  19. [27]

    Ava active speaker: An audio-visual dataset for active speaker detection,

    J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L. Kaver, S. Ramaswamy, A. Stopczynski, C. Schmid, Z. Xi et al., “Ava active speaker: An audio-visual dataset for active speaker detection,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Si...

    2020

  20. [28]

    An overview of deep-learning- based audio-visual speech enhancement and separation,

    D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y . Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning- based audio-visual speech enhancement and separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1368–1396, 2021

    2021

  21. [29]

    Audio and visual modality combination in speech processing appli- cations,

    G. Potamianos, E. Marcheret, Y . Mroueh, V . Goel, A. Koum- baroulis, A. Vartholomaios, and S. Thermos, “Audio and visual modality combination in speech processing appli- cations,” in The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Mod...

    2017

  22. [30]

    A review of on-device fully neural end-to-end automatic speech recognition algorithms,

    C. Kim, D. Gowda, D. Lee, J. Kim, A. Kumar, S. Kim, A. Garg, and C. Han, “A review of on-device fully neural end-to-end automatic speech recognition algorithms,” in 2020 54th Asilomar Conference on Signals, Systems, and Computers. IEEE, 2020, pp. 277–283. 16

    2020

  23. [31]

    Demucs-mobile: On-device lightweight speech enhance- ment

    L. Lee, Y . Ji, M. Lee, M.-S. Choi, and N. Coporation, “Demucs-mobile: On-device lightweight speech enhance- ment.” in Interspeech, 2021, pp. 2711–2715

    2021

  24. [32]

    Limitaccess: on-device tinyml based robust speech recognition and age classification,

    M. Maayah, A. Abunada, K. Al-Janahi, M. E. Ahmed, and J. Qadir, “Limitaccess: on-device tinyml based robust speech recognition and age classification,” Discover Artificial Intelligence, vol. 3, no. 1, p. 8, 2023

    2023

  25. [33]

    Deep learning based real-time speech enhancement for dual-microphone mobile phones,

    K. Tan, X. Zhang, and D. Wang, “Deep learning based real-time speech enhancement for dual-microphone mobile phones,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 1853–1863, 2021

    2021

  26. [34]

    Fast real-time personalized speech enhancement: End-to-end enhancement network (e3net) and knowledge distillation,

    M. Thakker, S. E. Eskimez, T. Yoshioka, and H. Wang, “Fast real-time personalized speech enhancement: End-to-end enhancement network (e3net) and knowledge distillation,” arXiv preprint arXiv:2204.00771 , 2022

    2022 arXiv

  27. [35]

    Cochleanet: A robust language-independent audio-visual model for real-time speech enhancement,

    M. Gogate, K. Dashtipour, A. Adeel, and A. Hussain, “Cochleanet: A robust language-independent audio-visual model for real-time speech enhancement,” Information Fusion, vol. 63, pp. 273–285, 2020

    2020

  28. [36]

    Lite audio-visual speech enhancement,

    S.-Y . Chuang, Y . Tsao, C.-C. Lo, and H.-M. Wang, “Lite audio-visual speech enhancement,” arXiv preprint arXiv:2005.11769, 2020

    2005 arXiv

  29. [37]

    Visual speech enhancement without a real visual stream,

    S. B. Hegde, K. Prajwal, R. Mukhopadhyay, V . P. Nambood- iri, and C. Jawahar, “Visual speech enhancement without a real visual stream,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2021, pp. 1926–1935

    2021

  30. [38]

    Smil: Multimodal learning with severely missing modality,

    M. Ma, J. Ren, L. Zhao, S. Tulyakov, C. Wu, and X. Peng, “Smil: Multimodal learning with severely missing modality,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 3, 2021, pp. 2302–2310

    2021

  31. [39]

    Towards good practices for missing modality robust action recognition,

    S. Woo, S. Lee, Y . Park, M. A. Nugroho, and C. Kim, “Towards good practices for missing modality robust action recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 3, 2023, pp. 2776–2784

    2023

  32. [40]

    Are multimodal transformers robust to missing modality?

    M. Ma, J. Ren, L. Zhao, D. Testuggine, and X. Peng, “Are multimodal transformers robust to missing modality?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 177–18 186

    2022

  33. [41]

    Mul- timodal prompting with missing modalities for visual recognition,

    Y .-L. Lee, Y .-H. Tsai, W.-C. Chiu, and C.-Y . Lee, “Mul- timodal prompting with missing modalities for visual recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 943–14 952

    2023

  34. [42]

    Multi- modality associative bridging through memory: Speech sound recollected from face video,

    M. Kim, J. Hong, S. J. Park, and Y . M. Ro, “Multi- modality associative bridging through memory: Speech sound recollected from face video,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 296–306

    2021

  35. [43]

    Speech reconstruction with reminiscent sound via visual voice memory,

    J. Hong, M. Kim, S. J. Park, and Y . M. Ro, “Speech reconstruction with reminiscent sound via visual voice memory,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3654–3667, 2021

    2021

  36. [44]

    Cromm-vsr: Cross-modal memory augmented visual speech recognition,

    M. Kim, J. Hong, S. J. Park, and Y . M. Ro, “Cromm-vsr: Cross-modal memory augmented visual speech recognition,” IEEE Transactions on Multimedia , vol. 24, pp. 4342–4355, 2021

    2021

  37. [45]

    Learn- ing audio-visual speech representation by masked multi- modal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learn- ing audio-visual speech representation by masked multi- modal cluster prediction,” arXiv preprint arXiv:2201.02184, 2022

    2022 arXiv

  38. [46]

    u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality,

    W.-N. Hsu and B. Shi, “u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality,” Advances in Neural Information Processing Systems, vol. 35, pp. 21 157–21 170, 2022

    2022

  39. [47]

    Time-domain transformer-based audiovisual speaker sepa- ration,

    V . A. Kalkhorani, A. Kumar, K. Tan, B. Xu, and D. Wang, “Time-domain transformer-based audiovisual speaker sepa- ration,” in Proc. INTERSPEECH, 2023, pp. 3472–3476

    2023

  40. [48]

    Attentive fusion enhanced audio-visual encoding for transformer based robust speech recognition,

    L. Wei, J. Zhang, J. Hou, and L. Dai, “Attentive fusion enhanced audio-visual encoding for transformer based robust speech recognition,” in 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 2020, pp. 638–643

    2020

  41. [49]

    Cross-attentional audio-visual fusion for weakly-supervised action localiza- tion,

    J.-T. Lee, M. Jain, H. Park, and S. Yun, “Cross-attentional audio-visual fusion for weakly-supervised action localiza- tion,” in International conference on learning representa- tions, 2020

    2020

  42. [50]

    Audio-visual fusion for emotion recognition in the valence-arousal space using joint cross-attention,

    R. G. Praveen, P. Cardinal, and E. Granger, “Audio-visual fusion for emotion recognition in the valence-arousal space using joint cross-attention,” IEEE Transactions on Biometrics, Behavior, and Identity Science , 2023

    2023

  43. [51]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning . pmlr, 2015, pp. 448–456

    2015

  44. [52]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

  45. [53]

    Easycom: An augmented reality dataset to support algorithms for easy communication in noisy environments,

    J. Donley, V . Tourbabin, J.-S. Lee, M. Broyles, H. Jiang, J. Shen, M. Pantic, V . K. Ithapu, and R. Mehra, “Easycom: An augmented reality dataset to support algorithms for easy communication in noisy environments,” arXiv preprint arXiv:2107.04174, 2021

    2021 arXiv

  46. [54]

    Interspeech 2021 deep noise suppression challenge,

    C. K. Reddy, H. Dubey, K. Koishida, A. Nair, V . Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srini- vasan, “Interspeech 2021 deep noise suppression challenge,” arXiv preprint arXiv:2101.01902 , 2021

    2021 arXiv

  47. [55]

    Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement,

    W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y . Adi, “Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement,” arXiv preprint arXiv:2212.11377, 2022

    2022 arXiv

  48. [56]

    Lip to speech synthesis with visual context attentional gan,

    M. Kim, J. Hong, and Y . M. Ro, “Lip to speech synthesis with visual context attentional gan,” Advances in Neural Information Processing Systems , vol. 34, 2021

    2021

  49. [57]

    Svts: Scalable video-to-speech synthesis,

    R. Mira, A. Haliassos, S. Petridis, B. W. Schuller, and M. Pantic, “Svts: Scalable video-to-speech synthesis,” arXiv preprint arXiv:2205.02058, 2022

    2022 arXiv

  50. [58]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  51. [59]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  52. [60]

    Sgdr: Stochastic gradient de- scent with warm restarts,

    I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient de- scent with warm restarts,” arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  53. [61]

    A short-time objective intelligibility measure for time- frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short-time objective intelligibility measure for time- frequency weighted noisy speech,” in 2010 IEEE interna- tional conference on acoustics, speech and signal processing. IEEE, 2010, pp. 4214–4217. 17

    2010

  54. [62]

    Sdr–half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 626–630

    2019

  55. [63]

    Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,

    A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. N...

    2001

  56. [64]

    Fullsubnet: A full-band and sub-band fusion model for real-time single- channel speech enhancement,

    X. Hao, X. Su, R. Horaud, and X. Li, “Fullsubnet: A full-band and sub-band fusion model for real-time single- channel speech enhancement,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6633–6637

    2021

  57. [65]

    Speech denoising in the waveform domain with self-attention,

    Z. Kong, W. Ping, A. Dantrey, and B. Catanzaro, “Speech denoising in the waveform domain with self-attention,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 7867–7871

    2022

  58. [66]

    Real time speech enhancement in the waveform domain,

    A. Defossez, G. Synnaeve, and Y . Adi, “Real time speech enhancement in the waveform domain,” arXiv preprint arXiv:2006.12847, 2020

    2006 arXiv

  59. [67]

    Weighted speech distortion losses for neural- network-based real-time speech enhancement,

    Y . Xia, S. Braun, C. K. Reddy, H. Dubey, R. Cutler, and I. Tashev, “Weighted speech distortion losses for neural- network-based real-time speech enhancement,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, p...

    2020

  60. [68]

    The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,

    C. K. Reddy, V . Gopal, R. Cutler, E. Beyrami, R. Cheng, H. Dubey, S. Matusevych, R. Aichner, A. Aazami, S. Braun et al. , “The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,” in INTERSPEECH, 2020

    2020

  61. [69]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu et al. , “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100 , 2020

    2005 arXiv

  62. [70]

    Transformer-based video front-ends for audio-visual speech recognition,

    D. Serdyuk, O. Braga, and O. Siohan, “Transformer-based video front-ends for audio-visual speech recognition,” arXiv preprint arXiv:2201.10439, 2022

    2022 arXiv

  63. [71]

    Demand: a collection of multi-channel recordings of acoustic noise in diverse environments,

    J. Thiemann, N. Ito, and E. Vincent, “Demand: a collection of multi-channel recordings of acoustic noise in diverse environments,” in Proc. Meetings Acoust , 2013, pp. 1–6

    2013

  64. [72]

    Out of time: automated lip sync in the wild,

    J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” in Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13. Springer, 2017, pp. 251–263

    2016

  65. [73]

    Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection,

    R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection,” in Proceedings of the 29th ACM international conference on multimedia, 2021, pp. 3927–3935

    2021

  66. [74]

    Learning long-term spatial-temporal graphs for active speaker detection,

    K. Min, S. Roy, S. Tripathi, T. Guha, and S. Majumdar, “Learning long-term spatial-temporal graphs for active speaker detection,” in European Conference on Computer Vision. Springer, 2022, pp. 371–387

    2022

  67. [75]

    Advancing active speaker detection for egocentric videos

    J. Huh, J. A. Ortiz, A. Kumar, A. Pandey, A. Aroudi, D. D. Wong, F. Nesta, B. Xu, and J. Donley, “Advancing active speaker detection for egocentric videos.” IEEE, 2025

    2025

  68. [76]

    Return of the devil in the details: Delving deep into convolutional nets,

    K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” arXiv preprint arXiv:1405.3531 , 2014

    2014 arXiv

  69. [77]

    Deep speaker embeddings for short-duration speaker verification

    G. Bhattacharya, M. J. Alam, and P. Kenny, “Deep speaker embeddings for short-duration speaker verification.” in Interspeech, 2017, pp. 1517–1521

    2017

  70. [78]

    V ox- celeb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V ox- celeb2: Deep speaker recognition,” arXiv preprint arXiv:1806.05622, 2018

    2018 arXiv

  71. [79]

    A stochastic approximation method,

    H. Robbins and S. Monro, “A stochastic approximation method,” The annals of mathematical statistics , pp. 400– 407, 1951

    1951

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.