Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-level mixture-of-experts router lets audio-visual speech recognition scale to a billion parameters while activating only about half of them.

desk verdict A solid AVSR MoE decoder with an honest ablation suite, but the headline adaptive-routing gain is measured against an untuned hard-routing baseline and single-seed runs. read the letter →

arxiv 2502.10447 v2 pith:PHCBCBNQ submitted 2025-02-11 eess.AS cs.CLcs.LG

Sungnyun Kim , Kangwook Jang , Sangmin Bae , Sungwoo Cho , Se-Young Yun This is my paper · ORCID
classification eess.AScs.CLcs.LG
keywords audio-visualspeechrecognitionmixtureofexpertshierarchicalgatingloadbiasinglossrobustmultilingualLRS3benchmarkMuAViC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that audio-visual speech recognition (AVSR) models can be scaled up in capacity without a proportional rise in compute if the extra parameters are organized into modality-specific expert groups and a learned router decides, token by token, how much each group contributes. The proposed model, MoHAVE, splits its Mixture-of-Experts layers into an audio group and a visual group, with an inter-modal router and a load biasing loss that teach the groups to specialize. On the noisy LRS3 benchmark it reports an average N-WER of 4.5% for its large configuration, below the 5.7% of the dense AV-HuBERT-LARGE baseline while activating 553M of its 1.0B parameters. If these results hold, the recipe matters because it offers a practical way to make AVSR larger and more noise-tolerant without proportional computational cost, and it locates the gain in group-level routing rather than in expert count.

What carries the argument

The load-bearing mechanism is the two-level gating structure. An inter-modal router computes $q_i(x)=\operatorname{softmax}(V_r x)_i$ over the two expert groups and selects the top-$m$ groups; within each selected group an intra-modal router, $[W^A_r, W^V_r]x$, picks the top-1 expert, so the output is a probability-weighted sum of one audio expert and one visual expert. The group-level load biasing loss, $\mathcal{L}_S=(1-g^A_1 Q^A_1)+(1-g^V_2 Q^V_2)$, trains the inter-modal router to send audio-only tokens to the audio group and video-only tokens to the visual group, and it is combined with the standard load balancing loss and z-loss. What this machinery does is turn expert selection into a learned, per-token decision about modality trust, rather than a fixed hand-set assignment.

What would settle it

Re-run the LRS3 noise benchmark with the hard-routing baseline using a per-noise-type tuned fixed audio weight (for babble, the paper's Figure 4(b) suggests 0.3) instead of the fixed 0.5, and compare average N-WER against MoHAVE-LARGE's 4.5%; if the tuned fixed baseline matches or beats it, the advantage attributed to learned hierarchical routing would not hold.

Watch

Extended reading notes

Core claim

The central claim is that the obstacle to scaling AVSR is not capacity but routing: adding experts without structure gives little, while dividing experts into audio and visual groups and letting a hierarchical router balance them unlocks most of the gain. MoHAVE uses an inter-modal router to assign each token a weight for the audio group and the visual group, then intra-modal routers pick the top-1 expert inside each selected group. A load biasing loss supervises this split by pushing audio-only tokens to the audio group and video-only tokens to the visual group, while audio-visual tokens are left free. The paper reports that the learned split tracks the noise: under babble noise and low SNRs the model shifts tokens to the visual group, and under cleaner conditions it leans on audio. In Table 2, MoHAVE-LARGE reaches 4.5% average N-WER on noisy LRS3 versus 5.7% for AV-HuBERT-LARGE and 4.8% for hard routing, at 553M active parameters out of 1.0B, and removing the load biasing loss makes the gain largely disappear.

Load-bearing premise

The comparison that carries the claimed gain is against a hard-routing baseline that gives audio and visual experts equal weight; if that baseline were tuned to its best fixed weight per noise type, the gap MoHAVE's learned router reports would likely shrink, and the reported numbers come from single runs without error bars.

Editorial extensions

If this is right

  • MoHAVE-LARGE reports 4.5% average N-WER on LRS3 under babble, speech, music, and natural noise, improving on the 5.7% of AV-HuBERT-LARGE while activating 553M of 1.0B parameters.
  • The ablation without load biasing drops to 5.6% N-WER, close to the plain MoE baseline, so the biasing loss is what produces group specialization.
  • On the MuAViC multilingual benchmark, MoHAVE-LARGE reaches 37.4% average WER and 19.5 average BLEU, improving over the mA V-HuBERT re-implementation at a comparable active-parameter budget.
  • Combining MoHAVE's decoder with the CMA self-supervised training yields 4.2% average N-WER, the best number in the paper's comparison table.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own Figure 4(b) implies a cheaper competitor: a hard-routing baseline with a per-noise-type tuned fixed split (e.g., $p_A=0.3$ for babble) might close much of the gap, so the value of learned routing should be measured against tuned fixed routing, not only the 0.5 default.
  • Because the clean-environment multilingual results in Appendix D.2 show little or no improvement, the mechanism's benefit appears specific to noisy conditions; a testable prediction is that MoHAVE's gain over dense baselines grows monotonically as test SNR decreases.
  • The appendix reports language-dependent expert loads (Arabic tokens leaning visual, Spanish and French tokens leaning audio), suggesting the router may learn implicit language or phoneme priors; probing router weights on language- and noise-labeled data could reveal whether the mechanism transfers across languages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MoHAVE, a sparse Mixture-of-Experts decoder for audio-visual speech recognition with a two-level routing scheme: an inter-modal router decides weights between an audio expert group and a visual expert group, and intra-modal routers select one expert within each group. A load-biasing loss (Eq. 14) encourages audio-only tokens to use the audio group and video-only tokens to use the visual group, while audio-visual tokens are routed dynamically. On the noisy LRS3 benchmark, MoHAVE-LARGE reports N-WER 4.5%, compared with 5.7% for AV-HuBERT-LARGE and 4.8% for a hard-routing MoE, with 553M active parameters out of 1.0B. The paper also reports multilingual ASR and speech-to-text translation results on MuAViC and includes extensive ablations, including an honest appendix showing no clean-speech gains.

Significance. If the reported gains are robust, MoHAVE provides a useful recipe for scaling AVSR decoders: modality-specialized expert groups plus a learned inter-modal router improve noise robustness at substantially lower active-parameter cost than a dense large model. The internal ablations are informative: the (–) Load Biasing rows in Table 2 degrade sharply, and Appendix D reports clean multilingual and unimodal results honestly. However, the distinctive claim—that learned hierarchical routing outperforms hard routing—is currently supported by a single, untuned hard-routing baseline and by single-run numbers, so the significance of the hierarchical-gating contribution is not yet established at the level claimed in the abstract.

major comments (3)
  1. [§3.2, Figure 4(b), Table 2] The hard-routing baseline evaluates audio-visual tokens with a fixed equal weight pA=pV=0.5, yet Figure 4(b) shows that the optimal fixed weight is pA=0.3 under babble noise and pA≥0.5 under natural noise. Because Table 2 is reported per noise type, a hard-routing control with a per-noise-type fixed weight chosen on a validation split is feasible and should be reported. If this tuned fixed-weight hard router approaches or matches MoHAVE's N-WER (4.5% vs 4.8% for LARGE), the claimed advantage of per-token inter-modal routing is not demonstrated; the remaining benefit would reduce to a noise-dependent scalar. This is the most load-bearing weakness for the paper's central novelty.
  2. [Tables 2, 4, 5 and Abstract] All comparisons are single training runs without confidence intervals. The headline gap between MoHAVE-LARGE and the hard-routing baseline is 0.3 N-WER (4.5 vs 4.8), and many per-SNR gaps in Table 5 are below 0.2 WER. Small differences of this size are within the typical run-to-run variability of large speech models, so the state-of-the-art claim needs either multiple seeds with reported variance or a significance test. This is especially important because the abstract makes a strong 'new standard' claim.
  3. [§5.4, Table 4] In the multilingual comparison, the hard-routing baseline is worse than the plain mA V-HuBERT baseline in both ASR (39.3 vs 38.9 WER) and translation (16.8 vs 18.5 BLEU). This suggests the hard-routing setup is suboptimal for the multilingual task, which complicates the interpretation of MoHAVE's multilingual gains over hard routing. Please explain this result or repair the baseline before drawing conclusions from Table 4.
minor comments (6)
  1. [§5.4, Eq. (14), Figure 4(a)] The expert-load-by-modality pattern for audio-only and video-only sequences is directly encouraged by the load-biasing loss, so it is a check of the training objective rather than evidence of emergent adaptivity. Please rephrase the text to avoid implying that this pattern emerges without the loss; the audio-visual-token analysis in Figure 5 is the relevant evidence for dynamic adaptation.
  2. [Abstract and Conclusion] The phrase 'setting a new standard' overstates the evidence given the small gain over the hard-routing baseline and the lack of clean-speech gains in Appendix Table 7; please temper the claim.
  3. [Table 4 / Appendix D.2] The paper should note explicitly in the main text that MoHAVE does not improve over the mA V-HuBERT baseline in clean multilingual tasks (Table 7); currently only the appendix reports this.
  4. [Section 4.2] The claim that audio-visual tokens are 'uniformly dispatched on average' is not derived from the losses in Eqs. (3)-(14). Please clarify what mechanism (e.g., initialization, entropy regularization) enforces this, or soften the claim.
  5. [Figure 4(b)] Please report the numerical WER values and, if available, error bars for the pA sweep; the current figure only supports a qualitative statement.
  6. [Reproducibility] No code or trained checkpoints are provided; releasing them would help verify the small differences in Table 2.

Circularity Check

1 steps flagged · score 3.0 of 10

Fig. 4(a) validates its own training objective by construction; the core benchmark comparison remains independent.

  1. self definitional [Section 4.2 Eq. (14) and Section 5.4 Figure 4(a)]
    "This load biasing loss encourages the inter-modal router to assign higher weights to EA experts for audio sequences and to EV experts for video sequences. ... LS = LAS + LVS = (1− gA1 · QA1) + (1− gV2 · QV2). ... when processing audio-only sequences, the model predominantly activates the audio expert group, while for video-only sequences, the visual expert group is mainly utilized. This distribution validates the effectiveness of our load biasing loss."

    Eq. (14) is exactly the objective that pushes gA1·QA1 toward 1 on audio-only tokens and gV2·QV2 toward 1 on video-only tokens. Therefore Figure 4(a)'s finding that audio-only inputs activate the audio expert group and video-only inputs activate the visual expert group is the trained behavior the loss was designed to produce; presenting it as validation of the load biasing loss restates the definition of LS rather than providing independent evidence. This does not by itself undermine the main N-WER comparisons, and the noise-adaptive shift for audio-visual tokens in Figure 5 is not forced by LS because audio-visual tokens are explicitly excluded from the load biasing loss.

full rationale

The only clear circular step is the expert-load-by-modality analysis in Figure 4(a), which is a self-definitional artifact of the load biasing loss: the loss directly rewards audio-only tokens selecting audio experts and video-only tokens selecting visual experts, so the observed load pattern is a restatement of the objective, not an emergent discovery. This step is supporting evidence rather than the central claim. The main benchmark results in Tables 2-4 are independent of this circularity: they compare full systems against external baselines and ablations, and the claimed advantage does not reduce to the load biasing loss by construction. The noise-adaptive routing for audio-visual tokens (Figure 5) is also not directly forced, since audio-visual tokens are excluded from the biasing loss, leaving genuine empirical content. The hard-routing baseline uses a fixed pA=0.5 even though the paper's own Figure 4(b) shows per-noise-type optimal fixed weights; this is a legitimate experimental-control concern that could weaken the adaptive-routing comparison, but it is not circularity. Self-citations such as the N-WER metric and CMA baseline are used as tools or baselines and are not load-bearing in a way that makes the derivation equal to its inputs. Overall, one minor self-definitional analysis exists, while the central performance claims retain independent content, warranting a score of 3.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The method introduces several tunable hyperparameters (loss weights, modality dropout, noise schedule, expert counts, training schedule) that the central performance claim depends on. The core domain assumptions are that sparse MoE training is stable with standard auxiliary losses, that the pretrained encoders are adequate, and that the evaluation protocol captures robustness. The main invented component is the inter-modal router, whose specialization behavior is partly a consequence of the load biasing loss rather than an independent discovery.

free parameters (5)
  • load biasing loss weight c_S = 1e-2
    Coefficient of L_S in Eq. (15); chosen by hand in Section 4.2, not inherited from prior work.
  • modality drop probability = 0.25
    Randomly drop audio or video with 25% probability during training for load biasing; Section 5.1.
  • noise corruption schedule = 25% of training set, SNR ~ N(0,5)
    Data augmentation for robust AVSR training; Section 5.1.
  • experts per group and total = 4 audio + 4 visual = 8
    Architecture choice in Section 5.1; top-1 per group to match top-2 MoE compute.
  • encoder freeze schedule = frozen for 90K, fine-tuned 30K of 120K steps
    Training schedule in Appendix A.2; chosen following Shi et al. (2022b).
assumptions (4)
  • domain assumption Sparse MoE with load balancing (Eq. 3) and z-loss (Eq. 6) trains stably and gives capacity without proportional compute.
    Adopted from Fedus et al. (2022) and Zoph et al. (2022); Section 3.1 and used by all MoE variants.
  • domain assumption The pretrained AV-HuBERT / AV2AV encoders provide sufficiently strong frozen audio-visual representations for the decoder improvements to transfer.
    All experiments build on these encoders (Section 5.1, Appendix A.2); gains are decoder-side only.
  • domain assumption The N-WER and noise-augmented evaluation protocol (MUSAN, LRS3 speech noise, SNR in {-10,...,10}) measures meaningful robustness.
    Protocol from Shi et al. (2022b) and the authors' own Kim et al. (2024); Section 5.1.
  • ad hoc to paper The hard-routing baseline should average both expert groups with equal weight pA=pV=0.5 for audio-visual tokens.
    Defined in Section 3.2 and used as comparison in Table 2; Figure 4(b) shows pA=0.3 is better under babble noise, so the baseline is not the best fixed policy.
invented entities (1)
  • Inter-modal router (hierarchical gating)
    purpose: Learns a per-token weight over audio and visual expert groups, enabling dynamic modality weighting in AVSR decoding.
    Introduced in Section 4.1 and trained with the load biasing loss; its noise-adaptive behavior is only evidenced by this paper's single-run experiments, and part of its behavior is directly enforced by the training loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition." pith.science (2026). https://pith.science/paper/PHCBCBNQ

@misc{pith2026250210447,
  author       = {Pith},
  title        = {Pith review of: MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PHCBCBNQ}},
  note         = {Machine review of arXiv:2502.10447}
}
read the original abstract

Audio-visual speech recognition (AVSR) has become critical for enhancing speech recognition in noisy environments by integrating both auditory and visual modalities. However, existing AVSR systems struggle to scale up without compromising computational efficiency. In this study, we introduce MoHAVE (Mixture of Hierarchical Audio-Visual Experts), a novel robust AVSR framework designed to address these scalability constraints. By leveraging a Mixture-of-Experts (MoE) architecture, MoHAVE activates modality-specific expert groups, ensuring dynamic adaptation to various audio-visual inputs with minimal computational overhead. Key contributions of MoHAVE include: (1) a sparse MoE framework that efficiently scales AVSR model capacity, (2) a hierarchical gating mechanism that dynamically utilizes the expert groups based on input context, enhancing adaptability and robustness, and (3) remarkable performance across robust AVSR benchmarks, including LRS3 and MuAViC transcription and translation tasks, setting a new standard for scalable speech recognition systems.

Figures

Figures reproduced from arXiv: 2502.10447 by the authors.

Figure 1
Figure 1. Comparison of AVSR models based on standard Trans￾formers (AV-HuBERT, Shi et al. 2022a), MoE, and MoHAVE, evaluated under babble noise. The MoE structure boosts the model capacity while maintaining the number of activations. MoHAVE-BASE (359M) achieves similar performance to AV￾HuBERT-LARGE (477M) while activating only 189M parameters. tages over audio-only automatic speech recognition (ASR) approaches. This multimo… view at source ↗
Figure 2
Figure 2. Overview of sparsely-gated MoE for AVSR. A select subset of experts are activated for each token representation (xt). where p˜ is the normalization of top-k probabilities. Note that each expert follows the same structure as a feed-forward network (FFN) in a Transformer block [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (b) visualizes the hard routing mechanism with au￾dio and visual expert groups. During training, audio or video sequence is randomly dropped, leading to subsets A and V within a batch B, consisting of audio-only or video-only sequences, respectively. A token representation xt ∈ A in￾dicates that the cross-attention module processes the input textt with Enc(a, 0)—where the visual component is zeroed out—and vice vers… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) Expert load distribution in MoHAVE according to input modalities, with expert selection frequencies weighted by the inter-modal router’s output probability. (b) Performance of the hard routing strategy under different weight assignments to audio expert group. The v…
Figure 5
Figure 5. Figure 5: Expert load distribution in MoHAVE for the audio group (solid bars) and visual group (dashed bars) across noisy audio-visual sequences under babble (left) and natural (right) noise. Full layer-wise results are provided in AppendixC.1. more tokens to the visual group, w…
Figure 6
Figure 6. Figure 6: Expert load distribution in MoHAVE for the audio group (solid bars) and visual group (dashed bars) across noisy audio-visual sequences under babble (first row) and natural (second row) noise. The frequency of each expert has been weighted by the inter-modal router’s ou…
Figure 7
Figure 7. Figure 7: Expert load distribution in multilingual AVSR MoHAVE for the audio group (solid bars) and visual group (dashed bars). 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

    eess.AS 2025-05 conditional novelty 4.0 of 10

    Llama-SMoP-DEDR, a sparse mixture of projectors with modality-specific experts and routers, lowers word error rate for LLM-based AVSR on LRS3, mainly with smaller LLMs.

Reference graph

Works this paper leans on

84 extracted references · 60 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    S., Senior, A., Vinyals, O., and Zisserman, A

    Afouras, T., Chung, J. S., Senior, A., Vinyals, O., and Zisserman, A. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 44 0 (12): 0 8717--8727, 2018 a

    2018

  4. [4]

    S., and Zisserman, A

    Afouras, T., Chung, J. S., and Zisserman, A. Lrs3-ted: a large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496, 2018 b

    arXiv 2018

  5. [5]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 4895--4901, 2023

    2023

  6. [6]

    Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation

    Anwar, M., Shi, B., Goswami, V., Hsu, W.-N., Pino, J., and Wang, C. Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation. In Interspeech. International Speech Communication Association, 2023

    2023

  7. [7]

    Xls-r: Self-supervised cross-lingual speech representation learning at scale

    Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., et al. Xls-r: Self-supervised cross-lingual speech representation learning at scale. In Interspeech. International Speech Communication Association, 2022

    work page 2022

  8. [8]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 0 12449--12460, 2020

    2020

Show all 84 references
  1. [9]

    C., Dale, D., Dong, N., Duquenne, P.-A., Elsahar, H., Gong, H., Heffernan, K., Hoffman, J., et al

    Barrault, L., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong, N., Duquenne, P.-A., Elsahar, H., Gong, H., Heffernan, K., Hoffman, J., et al. Seamlessm4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596, 2023

    2023 arXiv

  2. [10]

    and Timofte, R

    Burchi, M. and Timofte, R. Audio-visual efficient conformer for robust speech recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 2258--2267, 2023

    2023

  3. [11]

    Large language models are strong audio-visual speech recognition learners

    Cappellazzo, U., Kim, M., Chen, H., Ma, P., Petridis, S., Falavigna, D., Brutti, A., and Pantic, M. Large language models are strong audio-visual speech recognition learners. arXiv preprint arXiv:2409.12319, 2024

    2024 arXiv

  4. [12]

    Chen, C., Hu, Y., Zhang, Q., Zou, H., Zhu, B., and Chng, E. S. Leveraging modality-specific representations for audio-visual speech recognition via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 12607--12615, 2023

    2023

  5. [13]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing

    Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16 0 (6): 0 1505--1518, 2022

    2022

  6. [14]

    Mixtures of experts for audio-visual learning

    Cheng, Y., Li, Y., He, J., and Feng, R. Mixtures of experts for audio-visual learning. Advances in Neural Information Processing Systems, 2024

    2024

  7. [15]

    Self-supervised learning with random-projection quantizer for speech recognition

    Chiu, C.-C., Qin, J., Zhang, Y., Yu, J., and Wu, Y. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pp.\ 3915--3924. PMLR, 2022

    2022

  8. [16]

    J., Kim, M., and Ro, Y

    Choi, J., Park, S. J., Kim, M., and Ro, Y. M. Av2av: Direct audio-visual speech to audio-visual speech translation with unified audio-visual speech representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 27325--27337, 2024

    2024

  9. [17]

    S., Nagrani, A., and Zisserman, A

    Chung, J. S., Nagrani, A., and Zisserman, A. Voxceleb2: Deep speaker recognition. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, volume 2018, pp.\ 1086--1090, 2018

    2018

  10. [18]

    Unified scaling laws for routed language models

    Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. Unified scaling laws for routed language models. In International conference on machine learning, pp.\ 4057--4086. PMLR, 2022

    2022

  11. [19]

    Stablemoe: Stable routing strategy for mixture of experts

    Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., and Wei, F. Stablemoe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 7085--7095, 2022

    2022

  12. [20]

    A study of dropout-induced modality bias on robustness to missing video frames for audio-visual speech recognition

    Dai, Y., Chen, H., Du, J., Wang, R., Chen, S., Wang, H., and Lee, C.-H. A study of dropout-induced modality bias on robustness to missing video frames for audio-visual speech recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp....

    2024

  13. [21]

    and Luettin, J

    Dupont, S. and Luettin, J. Audio-visual speech modeling for continuous speech recognition. IEEE transactions on multimedia, 2 0 (3): 0 141--151, 2000

    2000

  14. [22]

    W., and Matt, P

    Elizabeth, S., Matthew, W., Jacob, B., Cattoni, R., Negri, M., Turchi, M., Oard, D. W., and Matt, P. The multilingual tedx corpus for speech recognition and translation. In Proceedings of Interspeech 2021, pp.\ 3655--3659, 2021

    2021

  15. [23]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  16. [24]

    Boosting speech recognition robustness to modality-distortion with contrast-augmented prompts

    Fu, D., Cheng, X., Yang, X., Wang, H., Zhao, Z., and Jin, T. Boosting speech recognition robustness to modality-distortion with contrast-augmented prompts. In ACM Multimedia, 2024

    2024

  17. [25]

    Conformer: Convolution-augmented transformer for speech recognition

    Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. Conformer: Convolution-augmented transformer for speech recognition. In Interspeech. International Speech Communication Association, 2020

    2020

  18. [26]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  19. [27]

    Jointly learning visual and auditory speech representations from raw data

    Haliassos, A., Ma, P., Mira, R., Petridis, S., and Pantic, M. Jointly learning visual and auditory speech representations from raw data. The Eleventh International Conference on Learning Representations, 2023

    2023

  20. [28]

    Braven: Improving self-supervised pre-training for visual and auditory speech recognition

    Haliassos, A., Zinonos, A., Mira, R., Petridis, S., and Pantic, M. Braven: Improving self-supervised pre-training for visual and auditory speech recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 11431--1143...

    2024

  21. [29]

    XLAVS - R : Cross-lingual audio-visual speech representation learning for noise-robust speech perception

    Han, H., Anwar, M., Pino, J., Hsu, W.-N., Carpuat, M., Shi, B., and Wang, C. XLAVS - R : Cross-lingual audio-visual speech representation learning for noise-robust speech perception. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024

  22. [30]

    Hong, J., Kim, M., Yoo, D., and Ro, Y. M. Visual context-driven audio feature enhancement for robust end-to-end audio-visual speech recognition. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, volume 2022, pp.\ 2838--...

    2022

  23. [31]

    Hong, J., Kim, M., Choi, J., and Ro, Y. M. Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18783--18794, 2023

    2023

  24. [32]

    and Shi, B

    Hsu, W.-N. and Shi, B. u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality. Advances in Neural Information Processing Systems, 35: 0 21157--21170, 2022

    2022

  25. [33]

    H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A

    Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 0 3451--3460, 2021

    2021

  26. [34]

    N., Zhang, Y., and Beaufays, F

    Hu, K., Li, B., Sainath, T. N., Zhang, Y., and Beaufays, F. Mixture-of-expert conformer for streaming multilingual asr. In Interspeech. International Speech Communication Association, 2023 a

    2023

  27. [35]

    Hu, Y., Chen, C., Li, R., Zou, H., and Chng, E. S. Mir-gan: Refining frame-level modality-invariant representations with adversarial network for audio-visual speech recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume ...

    2023

  28. [36]

    Hu, Y., Li, R., Chen, C., Qin, C., Zhu, Q.-S., and Chng, E. S. Hearing lips in noise: Universal viseme-phoneme mapping and transfer for robust audio-visual speech recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2023

  29. [37]

    Hu, Y., Li, R., Chen, C., Zou, H., Zhu, Q., and Chng, E. S. Cross-modal global interaction and local alignment for audio-visual speech recognition. In 32nd International Joint Conference on Artificial Intelligence, IJCAI 2023, pp.\ 5076--5084. International Joint Conferences o...

    2023

  30. [38]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 3 0 (1): 0 79--87, 1991

    1991

  31. [39]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  32. [40]

    Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6 0 (2): 0 181--214, 1994

    1994

  33. [41]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  34. [42]

    Learning video temporal dynamics with cross-modal attention for robust audio-visual speech recognition

    Kim, S., Jang, K., Bae, S., Kim, H., and Yun, S.-Y. Learning video temporal dynamics with cross-modal attention for robust audio-visual speech recognition. IEEE Spoken Language Technology Workshop (SLT), 2024

    2024

  35. [43]

    Multi-task corrupted prediction for learning robust audio-visual speech representation

    Kim, S., Cho, S., Bae, S., Jang, K., and Yun, S.-Y. Multi-task corrupted prediction for learning robust audio-visual speech representation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=WEQL5ksDnB

    2025

  36. [44]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  37. [45]

    Moai: Mixture of all intelligence for large language and vision models

    Lee, B.-K., Park, B., Won Kim, C., and Man Ro, Y. Moai: Mixture of all intelligence for large language and vision models. In European Conference on Computer Vision, pp.\ 273--302. Springer, 2025

    2025

  38. [46]

    \ GS \ hard: Scaling giant models with conditional computation and automatic sharding

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. \ GS \ hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview.net...

    2021

  39. [47]

    Unified cross-modal attention: Robust audio-visual speech recognition and beyond

    Li, J., Li, C., Wu, Y., and Qian, Y. Unified cross-modal attention: Robust audio-visual speech recognition and beyond. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 0 1941--1953, 2024 a

    1941

  40. [48]

    Pace: Unified multi-modal dialogue pre-training with progressive and compositional experts

    Li, Y., Hui, B., Yin, Z., Yang, M., Huang, F., and Li, Y. Pace: Unified multi-modal dialogue pre-training with progressive and compositional experts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 13402-...

    2023

  41. [49]

    Uni-moe: Scaling unified multimodal llms with mixture of experts

    Li, Y., Jiang, S., Hu, B., Wang, L., Zhong, W., Luo, W., Ma, L., and Zhang, M. Uni-moe: Scaling unified multimodal llms with mixture of experts. arXiv preprint arXiv:2405.11273, 2024 b

    2024 arXiv

  42. [50]

    Av-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations

    Lian, J., Baevski, A., Hsu, W.-N., and Auli, M. Av-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp.\ 1--8. IEEE, 2023

    2023

  43. [51]

    Moe-llava: Mixture of experts for large vision-language models

    Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Zhang, J., Ning, M., and Yuan, L. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024

    2024 arXiv

  44. [52]

    W., and Pantic, M

    Ma, P., Mira, R., Petridis, S., Schuller, B. W., and Pantic, M. Lira: Learning visual speech representations from audio through self-supervision. In Interspeech. International Speech Communication Association, 2021 a

    2021

  45. [53]

    End-to-end audio-visual speech recognition with conformers

    Ma, P., Petridis, S., and Pantic, M. End-to-end audio-visual speech recognition with conformers. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 7613--7617. IEEE, 2021 b

    2021

  46. [54]

    Auto-avsr: Audio-visual speech recognition with automatic labels

    Ma, P., Haliassos, A., Fernandez-Lopez, A., Chen, H., Petridis, S., and Pantic, M. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  47. [55]

    Recurrent neural network transducer for audio-visual speech recognition

    Makino, T., Liao, H., Assael, Y., Shillingford, B., Garcia, B., Braga, O., and Siohan, O. Recurrent neural network transducer for audio-visual speech recognition. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pp.\ 905--912. IEEE, 2019

    2019

  48. [56]

    Mm1: methods, analysis and insights from multimodal llm pre-training

    McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Belyi, A., et al. Mm1: methods, analysis and insights from multimodal llm pre-training. In European Conference on Computer Vision, pp.\ 304--323. Springer, 2025

    2025

  49. [57]

    Multimodal contrastive learning with limoe: the language-image mixture of experts

    Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., and Houlsby, N. Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35: 0 9564--9576, 2022

    2022

  50. [58]

    G., and Ogata, T

    Noda, K., Yamaguchi, Y., Nakadai, K., Okuno, H. G., and Ogata, T. Audio-visual speech recognition using deep learning. Applied intelligence, 42: 0 722--737, 2015

    2015

  51. [59]

    Leveraging unimodal self-supervised learning for multimodal audio-visual speech recognition

    Pan, X., Chen, P., Gong, Y., Zhou, H., Wang, X., and Lin, Z. Leveraging unimodal self-supervised learning for multimodal audio-visual speech recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 44...

    2022

  52. [60]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  53. [61]

    A call for clarity in reporting bleu scores

    Post, M. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp.\ 186--191, 2018

    2018

  54. [62]

    Lipsound2: Self-supervised pre-training for lip-to-speech reconstruction and lip reading

    Qu, L., Weber, C., and Wermter, S. Lipsound2: Self-supervised pre-training for lip-to-speech reconstruction and lip reading. IEEE transactions on neural networks and learning systems, 35 0 (2): 0 2772--2782, 2022

    2022

  55. [63]

    W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I

    Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp.\ 28492--28518. PMLR, 2023

    2023

  56. [64]

    Learning from the master: Distilling cross-modal advanced knowledge for lip reading

    Ren, S., Du, Y., Lv, J., Han, G., and He, S. Learning from the master: Distilling cross-modal advanced knowledge for lip reading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13325--13333, 2021

    2021

  57. [65]

    wav2vec: Unsupervised pre-training for speech recognition

    Schneider, S., Baevski, A., Collobert, R., and Auli, M. wav2vec: Unsupervised pre-training for speech recognition. In Interspeech. International Speech Communication Association, 2019

    2019

  58. [66]

    H., Nagrani, A., and Schmid, C

    Seo, P. H., Nagrani, A., and Schmid, C. Avformer: Injecting vision into frozen speech models for zero-shot av-asr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 22922--22931, 2023

    2023

  59. [67]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=B1ckMDqlg

    2017

  60. [68]

    Scaling vision-language models with sparse mixture of experts

    Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., and He, Y. Scaling vision-language models with sparse mixture of experts. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 11329--11344, 2023

    2023

  61. [69]

    Learning audio-visual speech representation by masked multimodal cluster prediction

    Shi, B., Hsu, W.-N., Lakhotia, K., and Mohamed, A. Learning audio-visual speech representation by masked multimodal cluster prediction. International Conference on Learning Representations, 2022 a

    2022

  62. [70]

    Robust self-supervised audio-visual speech recognition

    Shi, B., Hsu, W.-N., and Mohamed, A. Robust self-supervised audio-visual speech recognition. In Interspeech. International Speech Communication Association, 2022 b

    2022

  63. [71]

    Musan: A music, speech, and noise corpus

    Snyder, D., Chen, G., and Povey, D. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015

    2015 arXiv

  64. [72]

    The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings

    Thiemann, J., Ito, N., and Vincent, E. The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19. AIP Publishing, 2013

    2013

  65. [73]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  66. [74]

    T., and Li, H

    Wang, J., Pan, Z., Zhang, M., Tan, R. T., and Li, H. Restoring speaking lips from occlusion for audio-visual speech recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 19144--19152, 2024

    2024

  67. [75]

    Language-routing mixture of experts for multilingual and code-switching speech recognition

    Wang, W., Ma, G., Li, Y., and Du, B. Language-routing mixture of experts for multilingual and code-switching speech recognition. In Interspeech. International Speech Communication Association, 2023

    2023

  68. [76]

    R., and Hayashi, T

    Watanabe, S., Hori, T., Kim, S., Hershey, J. R., and Hayashi, T. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11 0 (8): 0 1240--1253, 2017

    2017

  69. [77]

    Robust audiovisual speech recognition models with mixture-of-experts

    Wu, Y., Peng, Y., Lu, Y., Chang, X., Song, R., and Watanabe, S. Robust audiovisual speech recognition models with mixture-of-experts. IEEE Spoken Language Technology Workshop (SLT), 2024

    2024

  70. [78]

    Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts

    You, Z., Feng, S., Su, D., and Yu, D. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech. International Speech Communication Association, 2021

    2021

  71. [79]

    Speechmoe2: Mixture-of-experts model with improved routing

    You, Z., Feng, S., Su, D., and Yu, D. Speechmoe2: Mixture-of-experts model with improved routing. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 7217--7221. IEEE, 2022

    2022

  72. [80]

    Visual hallucination elevates speech recognition

    Zhang, F., Zhu, Y., Wang, X., Chen, H., Sun, X., and Xu, L. Visual hallucination elevates speech recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 19542--19550, 2024

    2024

  73. [81]

    Self-supervised audio-visual speech representations learning by multimodal self-distillation

    Zhang, J.-X., Wan, G., Ling, Z.-H., Pan, J., Gao, J., and Liu, C. Self-supervised audio-visual speech representations learning by multimodal self-distillation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  74. [82]

    Uni-perceiver-moe: Learning sparse generalist models with conditional moes

    Zhu, J., Zhu, X., Wang, W., Wang, X., Li, H., Wang, X., and Dai, J. Uni-perceiver-moe: Learning sparse generalist models with conditional moes. Advances in Neural Information Processing Systems, 35: 0 2664--2678, 2022

    2022

  75. [83]

    Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning

    Zhu, Q., Zhou, L., Zhang, Z., Liu, S., Jiao, B., Zhang, J., Dai, L., Jiang, D., Li, J., and Wei, F. Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning. IEEE Transactions on Multimedia, 2023

    2023

  76. [84]

    St-moe: Designing stable and transferable sparse expert models

    Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.