{"id":"a08a4c59-a97d-4df4-961b-f567fcbb80e4","arxiv_id":"2502.01547","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"mWhisper-Flamingo reaches state-of-the-art multilingual WER on MuAViC and beats audio-only Whisper in noisy conditions across nine languages.","lead":"This paper presents mWhisper-Flamingo, a multilingual speech recognizer that combines an audio model (Whisper) with a lip-reading video model (AV-HuBERT) and a new training trick called decoder modality dropout. The model achieves state-of-the-art word error rates on a nine-language benchmark and shows clear gains over audio-only Whisper when speech is mixed with noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'all languages' noise-robustness claim is contradicted by Table II: mWhisper-Flamingo small does not improve Arabic WER at 0-dB babble.","rationale":"The reader's weakest assumption was that the multilingual AV-HuBERT visual encoder supplies useful lip features for all nine MuAViC languages, including low-resource Arabic, German, Greek, and Russian. Table II provides direct evidence that this assumption fails for Arabic in the small model, where the audio-visual model does not beat audio-only fine-tuned Whisper. This is the most load-bearing concern because the abstract's headline claim is exactly the statement that audio-visual mWhisper-Flamingo outperforms audio-only Whisper on all languages in noisy conditions, and that statement is false as written. The concern is concrete and internal to the paper, not a disagreement with consensus. The method still shows consistent average improvements and larger gains on higher-resource languages, and the ablation study supports the value of decoder modality dropout, so the paper is not fatally flawed. The appropriate outcome is a conditional acceptance requiring the authors to correct or scope the 'all languages' claim, add per-language significance testing, and clarify that the clean-condition SOTA is among audio-visual models rather than versus their own audio-only Whisper baseline. Since the reader's verdict was already CONDITIONAL, no change is needed.","tokens_in":14826,"tokens_out":3452,"duration_ms":31357,"concrete_test":"Download the released checkpoints from the provided GitHub repository and evaluate fine-tuned Whisper small and mWhisper-Flamingo small on the MuAViC Arabic test set with babble noise at 0-dB SNR, using the paper's text normalization and WER computation. Compute per-language WER with bootstrap confidence intervals over test utterances. If the Arabic WER difference is not statistically significant in favor of mWhisper-Flamingo, the abstract's 'all languages' claim must be weakened to 'most languages' or 'average across languages'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, as stated in the abstract and introduction, is that audio-visual mWhisper-Flamingo consistently outperforms audio-only Whisper on all languages in noisy conditions. Table II (0-dB babble) directly contradicts this for the small model: Arabic WER is 101 for both fine-tuned Whisper small and mWhisper-Flamingo small, with a reported relative improvement of -0.8%. For the medium model, Arabic improves only from 96.4 to 95.3 (1.1% relative), which is not a meaningful gain. Thus the 'all languages' claim fails on the lowest-resource language in the paper's own primary noisy benchmark. The multi-noise results in Figure 2 cover only Es, Fr, It, Pt, so 'all languages in noisy conditions' is an extrapolation beyond the data. Moreover, in clean conditions Table I shows mWhisper-Flamingo is slightly worse than fine-tuned audio-only Whisper (medium: 20.4 vs 20.1; small: 23.8 vs 23.5), so the 'state-of-the-art WER' claim must be scoped to audio-visual models. The load-bearing issue is not a stylistic quibble: the paper's headline promise of universal noise robustness is unsupported by its own table, and the failure appears exactly where the reader's weakest assumption predicted, namely low-resource languages where the multilingual visual encoder may not supply useful lip features.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes mWhisper-Flamingo, a multilingual audio-visual speech recognition model that combines Whisper with a multilingual AV-HuBERT visual encoder, using Flamingo-style gated cross-attention layers in the Whisper decoder. The key methodological addition is decoder modality dropout, which randomly zeroes either the audio or video embedding sequence at the decoder cross-attention input so the model trains on paired audio-visual, audio-only, and video-only inputs. Experiments on the 9-language MuAViC benchmark report clean-condition audio-visual SOTA results, a 0-dB babble average non-English WER improvement of about 10.4% relative for small and 10.6% for medium models over fine-tuned audio-only Whisper, multi-noise results for four higher-resource languages, and an ablation showing the dropout configuration and fine-tunable visual encoder are important. The paper releases code and models.","tokens_in":15120,"tokens_out":4336,"duration_ms":53826,"significance":"If the claims are scoped appropriately, the contribution is solid: it extends the Whisper-Flamingo approach to multilingual input, demonstrates a simple and transferable decoder-side modality dropout technique, and shows consistent average gains on the higher-resource MuAViC languages in challenging babble noise. The 0-dB babble results for Es, Fr, It, and Pt are internally consistent, and the ablation in Table III cleanly attributes the gain to the combination of a fine-tunable multilingual visual encoder and decoder modality dropout. The code and model release is a concrete reproducibility strength. The main weakness is that the abstract and introduction overclaim language coverage: the paper's own Table II shows no meaningful improvement for Arabic at 0-dB babble, so the 'all languages' noise-robustness statement is not supported as written. This is a correctness-of-claims issue rather than a methodological flaw, and it is fixable by re-scoping.","major_comments":[{"comment":"The abstract states that audio-visual mWhisper-Flamingo 'consistently outperforms audio-only Whisper on all languages in noisy conditions,' but Table II (0-dB babble, small models) shows Arabic WER of 101 for both fine-tuned Whisper small and mWhisper-Flamingo small, a relative change of -0.8%; for the medium model the Arabic gain is only 96.4 to 95.3, i.e., 1.1% relative. The only 9-language noisy result is the single babble condition in Table II, while the multi-noise, multi-SNR results in Figure 2 and Tables A1/A2 cover only Es, Fr, It, and Pt. The claim should be revised to state that the improvement is on average over non-English languages, or specifically for higher-resource languages, and the Arabic result should be discussed explicitly rather than hidden in an average.","section":"Abstract and Section I"},{"comment":"The clean-condition results do not support the unqualified 'state-of-the-art WER on MuAViC' statement. In Table I, fine-tuned audio-only Whisper medium achieves 20.1% average non-English WER, while mWhisper-Flamingo medium achieves 20.4%; for small models the numbers are 23.5% versus 23.8%. The video stream therefore provides no measurable clean-audio benefit in these experiments. The SOTA claim should be restricted to audio-visual models fine-tuned on MuAViC, and the paper should explicitly acknowledge that the reported video benefit appears only in noisy conditions.","section":"Section III-B and Table I"},{"comment":"The multi-noise, multi-SNR evidence is limited to four higher-resource languages (Es, Fr, It, Pt), but the conclusion claims the model 'outperforms audio-only Whisper in diverse noise settings' without the same language scope. Given the Arabic counterexample in Table II, the generalization from four languages to 'diverse noise settings' overall is not established. The authors should either report multi-noise results for low-resource languages or explicitly scope the conclusion to the four evaluated languages.","section":"Section III-C and Figure 2"}],"minor_comments":[{"comment":"The text says 'other dataloading details and hyperparameters closely follow Whisper-Flamingo [27]', but reference [27] is AV-CPL, not Whisper-Flamingo [17]. Either the citation is incorrect or it should cite [17].","section":"Section III-A"},{"comment":"There is a typo: 'the modal trains only on audio-visual inputs' should read 'the model trains only on audio-visual inputs'.","section":"Section III-D"},{"comment":"The definition of 'Avg non-En' is implicit. State explicitly that it is the arithmetic mean of the per-language WERs excluding English, and similarly for the H.R. and L.R. averages, to avoid ambiguity.","section":"Section III-B"},{"comment":"The caption states the WER is averaged over four languages and five SNR levels, but the aggregation is not fully specified. Clarify whether this is an average of per-language WERs or a pooled error rate, and add error bars if multiple evaluation runs are available.","section":"Figure 2"},{"comment":"The ablation is run on only five of the nine languages (En, Es, Fr, It, Pt), but the text refers to the result as 'the average non-En WER.' State the language subset explicitly in the table caption or in the surrounding text.","section":"Table III"},{"comment":"The paper does not specify how the MuAViC train, validation, and test splits are used, or that the validation set is used for checkpoint selection on the noisy condition. Adding this detail would improve reproducibility.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a signal processing letters submission, and the code and model release is a strength. The main concern is the overbroad 'all languages' noise-robustness claim, which is directly contradicted by the Arabic row of Table II. I recommend requiring a re-scoped abstract, an explicit discussion of the low-resource-language failure, and a restriction of the SOTA claim to audio-visual models before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the decoder modality dropout is the real contribution, and it works. The ablation on five languages drops average non-English WER at 0 dB babble from 44.6 to 36.6, a large and credible effect. The paper also ships code and models, and the benchmark comparisons are more careful than most letters.\n\nThe multilingual extension itself is straightforward—swap the English AV-HuBERT for a multilingual one and fine-tune everything—but the dropout trick is what makes it work, and the paper isolates that with a clean ablation. I also appreciate that they report per-language results and are honest that video does almost nothing in clean audio. The finding that video helps more in higher-resource languages is plausible and well documented.\n\nThe soft spots are real but not fatal. The abstract says audio-visual mWhisper-Flamingo \"consistently outperforms audio-only Whisper on all languages in noisy conditions,\" and Table II does not support that. Arabic gets the same 101 WER for both audio-only and audio-visual small models, a -0.8% relative change; the medium model gets 1.1%. So \"all languages\" is an overreach; the accurate claim is that video helps on average and on most languages. The Figure 2 multi-noise results cover only four languages, so that part is also extrapolated. They should fix the abstract and add a caveat about low-resource languages. There are also no error bars or multiple runs, which matters when claiming consistency across languages; the authors should report variance across seeds or justify why a single run is enough.\n\nThe stress-test concern about the visual encoder not helping low-resource languages is partly confirmed by Arabic, but it does not sink the paper. The average improvement still holds, and the mechanism is still doing something. It is a scope problem, not a wrong result.\n\nThis paper is for people working on AVSR or robust ASR. It is a solid engineering advance with a reusable technique. I would send it to peer review. Referees should ask for a corrected abstract, some measure of variability, and a discussion of the Arabic result. The core method and experiments are worth publishing.","headline":"Solid multilingual AVSR extension with a genuinely useful dropout trick, but the abstract overstates the 'all languages' noise-robustness claim; send to review with requested revisions.","tokens_in":15644,"tokens_out":2715,"would_cite":true,"duration_ms":22637,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that mWhisper-Flamingo, a late-fusion audio-visual model that trains Whisper's decoder on video-only and audio-visual inputs, beats audio-only Whisper on all nine MuAViC languages in noisy conditions and sets a new state…","keywords":["audio-visual speech recognition","multilingual speech recognition","noise robustness","decoder modality dropout","Whisper","AV-HuBERT","MuAViC","lip reading"],"falsifier":"Take the trained mWhisper-Flamingo model and decode each MuAViC language with the audio stream zeroed at test time, leaving video-only input. If on any lower-resource language, such as Arabic, German, Greek, or Russian, the video-only WER is close to the chance level for that language, or no better than the audio-only model's output, then the assumption that the multilingual lip encoder supports those languages is false and the claimed mechanism would not hold there.","tokens_in":14622,"feed_emoji":"👄","tokens_out":8075,"duration_ms":62374,"temperature":0.7,"pith_summary":"This paper sets out to show that multilingual audio-visual speech recognition can be built by attaching a pre-trained lip-reading video encoder to Whisper rather than training from scratch. The proposed model, mWhisper-Flamingo, combines Whisper's multilingual audio encoder and decoder with a multilingual AV-HuBERT visual encoder, and adds decoder modality dropout: during training, either both modalities or only the video stream is fed to the decoder, with the missing modality's features zeroed out. The authors claim this makes the decoder actually use lip information, and report that the audio-visual model consistently beats audio-only Whisper on all nine MuAViC languages in noisy conditions, with roughly a 10% relative improvement in average non-English WER at 0-dB babble. If true, this offers a practical recipe for making speech recognition robust to noise in languages that lack large audio-visual training sets.","feed_headline":"Lip-reading video cuts noisy multilingual speech errors by 10 percent","feed_subtitle":"Adding a multilingual lip-reading encoder to Whisper beats audio-only baselines under babble and other noise.","key_machinery":"The load-bearing mechanism is decoder modality dropout applied in a late-fusion decoder. Whisper's decoder already contains separate cross-attention layers for audio and video features; during training, the model randomly replaces one modality's embedding sequence by a zero vector, so the decoder must learn to transcribe from video-only input as well as from audio-visual input. The decisive configuration is to drop audio half the time ($p_V=0.5$) and never drop video ($p_A=0$), because Whisper was already fine-tuned on audio and the bottleneck is teaching the decoder to exploit the lip stream. The visual features come from a multilingual AV-HuBERT encoder pre-trained on multilingual videos, whose parameters are fine-tuned during the audio-visual stage, unlike the frozen encoder in the original Whisper-Flamingo.","core_discovery":"On the paper's own terms, the central discovery is that a late-fusion Whisper decoder can be taught to integrate visual speech by randomly zeroing out one modality's features during training. mWhisper-Flamingo replaces Whisper-Flamingo's English-only visual encoder with a multilingual AV-HuBERT, fine-tunes that encoder, and trains with probabilities $p_{AV}=0.5$, $p_A=0$, $p_V=0.5$, meaning half the steps present audio-visual inputs and half present video-only inputs. The ablation shows that without this dropout the average non-English WER on a five-language noisy subset is 44.6%, and with it the WER drops to 36.6%. At 0-dB babble on the full nine-language MuAViC benchmark, mWhisper-Flamingo small reaches 50.4% average non-English WER versus 55.3% for fine-tuned audio-only Whisper small, and the medium model reaches 43.7% versus 48.0%.","pith_inferences":["The same late-fusion dropout recipe could be applied to other multimodal systems, such as audio-visual translation or video question answering, whenever one modality is reliable and the other is informative but hard to learn from.","The result suggests a cheap route to noise-robust ASR for additional languages: keep Whisper's audio backbone and swap in a lip-reading encoder trained on that language's video, using video-heavy modality dropout, rather than collecting tens of thousands of hours of transcribed video.","Because video-only training mattered more than audio-only training, a testable prediction follows: mWhisper-Flamingo's advantage over audio-only Whisper should grow as SNR drops, and the model should decode from silent video alone with WER far below chance on every MuAViC language.","The fact that fine-tuning the visual encoder mattered more when dropout was enabled (36.6 versus 40.6) suggests the dropout itself creates the gradient pressure that makes visual fine-tuning useful; varying $p_V$ while keeping the encoder frozen would test this directly."],"forward_implications":["At 0-dB babble, mWhisper-Flamingo small cuts average non-English WER from 55.3% to 50.4% relative to fine-tuned audio-only Whisper small; the medium model cuts 48.0% to 43.7%.","The audio-visual model beats audio-only Whisper across six noise types and five SNR levels on Spanish, French, Italian, and Portuguese, with the largest gains on babble and overlapping speech.","A smaller audio-visual model (651M parameters) outperforms a larger audio-only model (769M parameters) on the higher-resource languages under 0-dB babble, 37.4% versus 38.1% WER.","On clean MuAViC audio, mWhisper-Flamingo medium achieves a non-English average WER of 20.4%, matching fine-tuned audio-only Whisper (20.1%) and beating all prior audio-visual models trained on the same 1,141 hours.","Ablations identify video-only dropout as the critical ingredient: removing it raises noisy non-English WER from 36.6% to 44.6%, while replacing it with audio-only dropout gives no improvement."],"supporting_citations":[{"why":"Defines the Whisper-Flamingo architecture and two-stage training that mWhisper-Flamingo extends, including the gated cross-attention layers and the babble-noise evaluation setup.","marker":"[17]"},{"why":"Supplies the multilingual AV-HuBERT visual encoder pre-trained on multilingual videos, which replaces the English-only encoder and is fine-tuned in this work.","marker":"[19]"},{"why":"Provides the MuAViC dataset, the nine-language benchmark and the clean/noisy evaluation protocol from which all results are reported.","marker":"[20]"},{"why":"Provides the original English AV-HuBERT representation learning method and the English-only encoder used as the comparison baseline in the ablation.","marker":"[18]"},{"why":"Supplies Whisper, the pre-trained multilingual audio encoder and text decoder that form the backbone of mWhisper-Flamingo.","marker":"[1]"},{"why":"Provides the XLAVS-R multilingual audio-visual model, the main baseline mWhisper-Flamingo is compared against as prior state of the art.","marker":"[45]"},{"why":"Supplies the noise types and SNR protocol used in the noisy evaluation, together with the AV-HuBERT baseline training recipe.","marker":"[13]"}],"fun_headline_variants":["Lip-reading video cuts noisy multilingual speech errors by 18%","Modality dropout: key to multilingual audio-visual speech recognition","mWhisper-Flamingo chips away at noisy multilingual WER with video","Audio-visual training with dropout beats audio-only Whisper in noise","Video lip-reading helps multilingual speech recognition resist babble"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if the multilingual AV-HuBERT lip encoder provides genuinely useful visual features for all nine MuAViC languages, including the four low-resource languages with as little as 10 hours of video; if the lip stream is uninformative for those languages, training the decoder half the time from video alone would hurt rather than help.","fun_headline_variants_meta":{"raw":{"variants":["Lip-reading video cuts noisy multilingual speech errors by 18%","Modality dropout: key to multilingual audio-visual speech recognition","mWhisper-Flamingo chips away at noisy multilingual WER with video","Audio-visual training with dropout beats audio-only Whisper in noise","Video lip-reading helps multilingual speech recognition resist babble"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000788,"raw_usage":{"total_tokens":3465,"prompt_tokens":923,"completion_tokens":2542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2453}},"tokens_in":539,"tokens_out":2542,"duration_ms":23134,"temperature":1.0,"reasoning_tokens":2453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:59:26.656479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained mWhisper-Flamingo model and decode each MuAViC language with the audio stream zeroed at test time, leaving video-only input. If on any lower-resource language, such as Arabic, German, Greek, or Russian, the video-only WER is close to the chance level for that language, or no better than the audio-only model's output, then the assumption that the multilingual lip encoder supports those languages is false and the claimed mechanism would not hold there.","supporting_citations":[{"cited_title":"Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation,","cited_arxiv_id":null,"evidence_quote":"Defines the Whisper-Flamingo architecture and two-stage training that mWhisper-Flamingo extends, including the gated cross-attention layers and the babble-noise evaluation setup."},{"cited_title":"Efficient training for multilingual visual speech recognition: Pre-training with discretized visual speech representation,","cited_arxiv_id":null,"evidence_quote":"Supplies the multilingual AV-HuBERT visual encoder pre-trained on multilingual videos, which replaces the English-only encoder and is fine-tuned in this work."},{"cited_title":"Muavic: A multilingual audio-visual corpus for robust speech recogni- tion and robust speech-to-text translation,","cited_arxiv_id":null,"evidence_quote":"Provides the MuAViC dataset, the nine-language benchmark and the clean/noisy evaluation protocol from which all results are reported."},{"cited_title":"Learning audio- visual speech representation by masked multimodal cluster prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the original English AV-HuBERT representation learning method and the English-only encoder used as the comparison baseline in the ablation."},{"cited_title":"Xlavs-r: Cross-lingual audio-visual speech representation learning for noise-robust speech perception,","cited_arxiv_id":null,"evidence_quote":"Provides the XLAVS-R multilingual audio-visual model, the main baseline mWhisper-Flamingo is compared against as prior state of the art."},{"cited_title":"Robust Self-Supervised Audio- Visual Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the noise types and SNR protocol used in the noisy evaluation, together with the AV-HuBERT baseline training recipe."}],"review_version":1}