{"id":"3890393f-b89b-44f3-ad66-488928ace4a7","arxiv_id":"2507.19037","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In a small multimodal language model for ASR, lip movements give the largest relative benefit at high noise while image/OCR context peaks at moderate noise, but the effect depends on architecture and input format.","lead":"This paper tests when extra inputs like lip movements, images, or OCR text improve speech-to-text accuracy in a multimodal language model, under different noise levels. It reports that synchronized visual inputs help most in loud noise, while unsynchronized context helps at moderate noise, and that these trends differ somewhat between Transformer and Mamba backbones.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3-Equations noise schedule (MUSAN on second half only) is a positional confound; the 'sweet spot' and lip-benefit trends may not persist under stationary noise.","rationale":"The reader's weakest assumption identified the representativeness of 3-Equations; I agree but sharpen it to a concrete positional artifact. The central assertion is a functional relationship between SNR and modality benefit. In 3-Equations, the corruption is not stationary: MUSAN noise occupies a fixed second half of each utterance. This gives the model a predictable clean region whose location interacts with input order and absolute positional encodings. Section VII-B explicitly invokes this mechanism to explain order effects, so the same mechanism can influence the O+A and L+A RB curves in Fig. 3. The peak of unsynchronized benefit at SNR=2.5 dB and the monotone increase of synchronized benefit may therefore encode properties of the noise scheduler rather than of modality synchrony. SlideA VSR, which uses full-utterance noise, shows unsynchronized modalities underperform the audio-only baseline at high noise, so the transfer is not established. A stationary-noise replication is a decisive check. The hyperparameter-transfer issue is real but secondary. Since the reader already recommended conditional acceptance and my concern supports that rather than overturning it, the verdict should remain unchanged.","tokens_in":17001,"tokens_out":4679,"duration_ms":51237,"concrete_test":"Construct a stationary-noise variant of 3-Equations: add MUSAN noise at the same nominal SNR over the entire utterance, and ideally a second variant with the noise segment appearing at random positions, while keeping the training SNR mix, model, and loss weights identical. Recompute the RB curves in Fig. 3 and Table I. If the OCR 'sweet spot' at moderate SNR disappears or shifts substantially, or if the lip RB no longer increases with noise, the central when-claim is an artifact of the second-half-only noise schedule. Also report the curves separately for audio-first and audio-later input orderings to separate the positional effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline 'when' claim—synchronized modalities help more as noise rises, unsynchronized modalities peak at moderate SNR—is identified from Fig. 3 and Table I on 3-Equations. The dataset's noise is added only to the second half of each utterance (Section IV-A), and the paper's own Section VII-B shows that the resulting clean first half can be located via absolute positional encodings depending on modality order. Thus the O+A RB peak at SNR=2.5 dB and the monotone increase of L+A RB are confounded by noise position and input ordering; they are not cleanly attributable to synchrony. SlideA VSR, which uses full-utterance noise, shows unsynchronized modalities actually hurt at high SNR (Table II), so the 3-Equations sweet spot cannot be assumed to transfer. The fact that loss weights are tuned on 3-Equations and Mamba needs different hyperparameters (Section VI) further weakens generalization, but the first issue is more load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a systematic empirical study of when and how supplementary visual and lip modalities improve MLLM-based ASR under acoustic noise. Using a synthetic 3-Equations dataset and two real-world corpora (SlideAVSR, S-MiT), it evaluates OPT- and Mamba-based DMLM models, computes WER, relative benefit (RB), and perceptual scores (PS) across SNR levels, and additionally studies visual representation quality, sequence noise, modality input order, and modality-weighted loss components. The central claims are that adding modalities usually helps ASR, synchronized modalities (lip movements) help most at high noise while unsynchronized modalities (image/OCR) peak at moderate noise, better visual representations help, Mamba exhibits similar multimodality trends to Transformers, and input format and loss weights significantly affect accuracy.","tokens_in":17205,"tokens_out":11379,"duration_ms":110834,"significance":"The study's controlled manipulation of SNR and the reported statistical window around minibatch shuffling are genuine strengths; the paper does not rely on a single arbitrary training run, and the comparisons across modality combinations are unusually systematic for this area. If the claims were fully supported, the paper would provide practical guidance for designing multimodal ASR front-ends and for choosing backbones, input encodings, and loss balancing. However, several of the abstract's global conclusions are not consistently supported by the paper's own tables, and the main noise-dependent 'when' result is derived from a synthetic corpus with a positional noise confound. The contribution nevertheless contains useful observations (e.g., sequence-noise effects, OCR benefits, negative results for raster and interleaved inputs) that deserve publication after substantial revision.","major_comments":[{"comment":"The central 'when' finding (Fig. 3 and Table I) is confounded by the 3-Equations noise schedule: MUSAN noise is added only to the second half of each utterance, and Section VII-B demonstrates that, with absolute positional encodings, the model can exploit the known location of the clean first half and that this location interacts with modality order. Because the synchronized-lip trend (increasing RB as SNR decreases) is the only controlled evidence for the abstract's claim (2), this trend may reflect the position of the clean half rather than the acoustic SNR itself. Please add a control experiment with stationary full-utterance noise on 3-Equations, or explicitly restrict the synchronized-modality claim to partial-noise conditions.","section":"Sections IV-A and VII-B"},{"comment":"The abstract's claim that 'higher-quality visual representations consistently improve ASR accuracy' is contradicted by the raster row of Table I: R+A has negative RB at every SNR (average -11.7%) despite raster being introduced as a 'perfect' 2D representation, and it is worse than both image and OCR. The explanation about sequence length and 2D positional encodings is plausible, but the blanket claim must be qualified to 'more abstract and informative representations' or to representations for which the model has adequate positional support, and the abstract should be revised accordingly.","section":"Section V-D, Table I; abstract claim (3)"},{"comment":"The abstract's claim that 'Mamba exhibits similar trends regarding the benefits of multimodality as do Transformers' is not supported by the reported numbers. On Mamba, I+A RB decreases as noise increases (from +10.8% at 20 dB to -2.4% at -20 dB), which the paper itself in Section VI says is 'different from the trend on OPT,' and L+A RB is negative at every SNR on Mamba. Moreover, Section VI reports that Mamba required different hyperparameters for different modality combinations, so using the 3-Equations-tuned lambda values for all models may unfairly penalize some Mamba configurations. The claim should be restricted to OCR-only or supported by per-combination hyperparameter tuning for each reported configuration.","section":"Section VI, Table IV; abstract claim (4)"},{"comment":"The abstract's causal framing that 'each modality provides complementary information' is in tension with the paper's own Perceptual Score analysis: Section V-B states that PS(I) is near zero for most SNR values and that the image's benefit 'might have more to do with regularization than complementary information.' If the image modality contributes mainly through regularization, the claim about complementary information should be reformulated, and the paper should provide a test that separates complementary-information benefits from regularization effects (e.g., an example where the image resolves an ambiguity that other modalities cannot resolve).","section":"Section V-B and Fig. 4; abstract claim (1)"}],"minor_comments":[{"comment":"The training-time sentence is garbled: 'for an I+A → T model, the training time of OPT grows to 305 sec/ep (55% increase), while OPT it increased to 643 sec/ep (68% increase)' appears to repeat 'OPT' where 'Mamba' is intended; please clarify which architecture corresponds to each number.","section":"Section VI"},{"comment":"The grid search over lambda_S, lambda_T, and lambda_I is not specified; please report the candidate values and the validation criterion used so that the tuning procedure is reproducible.","section":"Section III-C"},{"comment":"The sentence 'we find evidence that multimodal ASR can benefit from at least 4 modalities as inputs' is ambiguous; it should be reworded to refer to the 4-modality model or to the number of added modalities.","section":"Section V-A"},{"comment":"The 'blocked' input format is defined only by contrast with the interleaved format; please state explicitly whether 'blocked' means all audio tokens followed by all frame tokens (or vice versa) and whether the five video-frame token sequences are contiguous.","section":"Section VII-A"},{"comment":"The manuscript repeatedly refers to Supplementary Materials Sections I-VI for details that support main-text claims (e.g., the exact minibatch shuffling procedure); please ensure the supplementary material is available to reviewers and that the main text summarizes the key procedural details.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a legitimate extended version of the authors' own conference work, and the new experiments on Mamba, SlideAVSR, S-MiT, input ordering, and loss weights justify a journal submission. My recommendation of major revision is driven by the gap between the abstract's blanket claims and the evidence in the paper's own tables, especially the raster result, the Mamba image trend, and the positional noise confound in the central 3-Equations analysis. These are fixable within the scope of the manuscript, but they require additional experiments or careful rephrasing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a worthwhile extended ablation study of MLLM-based ASR, but the abstract claims more than the tables deliver, and the main 'when' result on 3-Equations is partly confounded by the noise schedule. Still worth sending to reviewers after the authors temper the claims.\n\nWhat's new: the Mamba-vs-Transformer comparison, ViT-vs-DALL-E encoders, interleaved vs blocked input, modality order swaps, and loss-weight ablations. The significance-window analysis using minibatch reshuffling is a nice touch, and the perceptual score distinction (complementary info vs regularization) is thoughtful. The SlideAVSR and S-MiT results add real-world evidence.\n\nWhere it wobbles. First, the abstract's claim that 'higher-quality visual representations consistently improve' is directly contradicted by the raster condition: a perfect 2-D representation performs worse than OCR or image. The authors give plausible reasons (token length, absolute positional encodings), but 'consistently' is wrong. Second, 'Mamba exhibits similar trends' is also not supported by Table IV: image RB on Mamba decreases with noise, opposite to OPT, and lip RB is negative at every SNR. The paper does acknowledge this in Section VI, so the abstract is simply overbroad. Third, and more load-bearing, the 3-Equations noise is added only to the second half of each utterance. That lets models use absolute position to locate the clean first half. Section VII-B explicitly makes this argument for the input-order results, but the same confound applies to the 'sweet spot' and 'lips help more as noise rises' curves. SlideAVSR, with full-utterance noise, does show a sweet spot for unsynchronized modalities, so the idea isn't baseless; but the synchronized-modality trend rests almost entirely on 3-Equations, and Mamba's lip result is a counterexample.\n\nThe cross-architecture loss-weight transfer is a lesser issue: lambdas tuned on 3-Equations are used for all models, and the authors report Mamba needs different hyperparameters. That weakens the Mamba comparison more than they let on.\n\nWho this is for: people building MLLM-based ASR and deciding how to order and weigh inputs. The paper is a serious empirical contribution, worth a real referee. My recommendation: send it out, but require the abstract be aligned with the actual numbers and the positional confound be either addressed with a stationary-noise experiment or heavily caveated. Also ask for code/data release.","headline":"A solid extended ablation of multimodal ASR, but the abstract overclaims two findings that the paper's own tables undercut, and the 3-Equations noise schedule confounds the headline 'when' result.","tokens_in":17715,"tokens_out":4245,"would_cite":false,"duration_ms":40362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that in multimodal LLM speech recognition, synchronized modalities like lip movements help most as noise rises, while unsynchronized image/OCR context helps most at moderate noise.","keywords":["multimodal large language models","automatic speech recognition","audio-visual speech recognition","noise robustness","lip movements","visual context","Mamba state space models","modality fusion"],"falsifier":"Run the same DMLM audio+OCR and audio+lips models on a real dataset with per-utterance SNR labels and measure relative benefit at SNR values from 20 to -20 dB: if OCR/image benefit does not peak at moderate SNR, or if lip benefit does not rise as SNR falls, the central claim is wrong.","tokens_in":16831,"feed_emoji":"🎙️","tokens_out":7486,"duration_ms":74196,"temperature":0.7,"pith_summary":"The paper is trying to establish that multimodality is not uniformly beneficial for LLM-based speech recognition: the value of each extra input channel is governed by how well it is synchronized with the audio and by the acoustic noise level. Specifically, lip movements, which track the speech signal in time, become more useful as SNR decreases, whereas static image or OCR context is most useful when noise is moderate—strong enough that audio alone is unreliable, weak enough that the model can still align the speech to the context. The authors support this with a discrete-token multimodal language model (DMLM) on a synthetic equations-reading dataset and two real-world video datasets, and they additionally report that better visual representations and encoders improve ASR, Mamba backbones follow the same qualitative trends, and input order and loss weights matter. If correct, the work turns 'add more modalities' into a design rule: choose modalities by noise regime, not by count.","feed_headline":"Synchronized cues beat image context as noise rises in multimodal ASR","feed_subtitle":"New tests on synthetic and real video show which extra input pays off at which noise level.","key_machinery":"The load-bearing object is DMLM, a discrete-token multimodal decoder-only language model: frozen modality encoders convert audio, lips, and images into token sequences that are concatenated and processed by an OPT-125M backbone trained with a modality-weighted cross-entropy loss. The paper's main analytical instruments are the relative benefit (RB) and perceptual score (PS) metrics, which quantify how much a modality adds or accounts for in WER terms. Two design manipulations carry the argument: varying SNR with MUSAN noise so that synchronized versus unsynchronized alignment can be compared, and moving visual input along a quality continuum from raw image to OCR to oracle-OCR.","core_discovery":"On its own terms, the paper's central discovery is a noise-dependent ordering of visual modality benefit in MLLM-based ASR. Using DMLM—a frozen-encoder decoder-only OPT-125M that tokenizes speech via Seamless, lips via AV-HuBERT, and images via DALL-E or ViT—the authors vary the input modalities on the 3-Equations synthetic corpus and measure relative benefit RB=(WER(A)−WER(A∪S))/WER(A). They find that adding OCR context to audio yields its highest RB at moderate SNR (about 33.5% at 2.5 dB, versus 12% at 10 dB and 9% at -20 dB), while adding synchronized lip movements yields an RB that is negative or near zero at high SNR and grows to about +10% at -20 dB. They also find that adding a fourth modality is not always better at low noise, that higher-quality visual representations (oracle OCR vs raw image) and stronger visual encoders improve accuracy, that Mamba-130M reproduces the OCR/lip trends but with lower accuracy and more hyperparameter sensitivity, and that audio-first input ordering and nonzero modality loss weights help. Similar unsynchronized-context trends appear on SlideAVSR with slide and OCR inputs.","pith_inferences":["One implication left implicit by the paper is a noise-adaptive inference policy: estimate per-utterance SNR, then route to image/OCR context at moderate noise and to lip input at high noise; the paper measures benefit per SNR bucket but does not build such a controller.","Since the authors attribute the 4-modality model's failure partly to sequence noise, I infer that compressing or filtering context tokens before concatenation should restore much of the multimodal gain at low SNR; a direct test would compare OCR with token pruning at SNR -10 dB.","The ViT-versus-DALL-E gap suggests semantic visual encoders aligned with language are more valuable for MLLM ASR than structural encoders; I would test this by scaling each encoder while holding token count fixed."],"forward_implications":["Adding more modalities to an MLLM ASR model is not universally helpful: the benefit is positive on average, but at low noise the 4-modality model underperforms 3-modality ones, so context length should be treated as a cost.","For unsynchronized context such as slides, images, or OCR, deployment is most valuable at moderate noise; at very low SNR the context no longer aligns with the speech and can hurt accuracy.","For synchronized lip input, the value grows with noise, so audio-visual front-ends are most justified in high-noise settings.","Improving visual encoders, especially their semantic quality, pays off directly in ASR accuracy even when the decoder is unchanged.","Mamba can reproduce the qualitative modality trends while training much faster, but its sensitivity to hyperparameters means tuned settings from a Transformer model should not be assumed to transfer."],"supporting_citations":[{"why":"Supplies the 3-Equations dataset and the initial DMLM-based modality analysis that this paper extends; the synthetic dataset is the main platform for the noise-dependent claims.","marker":"[1]"},{"why":"Defines DMLM, the discrete-token multimodal OPT-125M model and its pretraining and multimodal loss recipe that all experiments inherit.","marker":"[5]"},{"why":"Provides the frozen AV-HuBERT encoder that turns lip movements into discrete tokens, i.e., the synchronized modality at the center of the high-noise claim.","marker":"[44]"},{"why":"Provides SlideAVSR, the real-world slide-explanation dataset used to check the unsynchronized OCR/image trend and FQ Ranker filtering.","marker":"[49]"},{"why":"Provides Spoken Moments in Time, the real-world video dataset used for image/video and visual-encoder comparisons.","marker":"[50]"},{"why":"Supplies the MUSAN noise corpus used to create the SNR conditions that define the sweet-spot and high-noise regimes.","marker":"[52]"},{"why":"Defines the Mamba architecture used as the alternative backbone in the architecture-comparison experiments.","marker":"[17]"},{"why":"Defines the Perceptual Score metric used to attribute WER influence to individual modalities in the 4-modality model.","marker":"[37]"},{"why":"Earlier evidence that synchronized audio-visual benefit grows with noise; the paper's lip-movement trend is an extension of this.","marker":"[55]"}],"fun_headline_variants":["Noise decides which extra modality helps MLLM speech recognition","Lip sync aids at high noise, image context at moderate SNR","Multimodal ASR: audio-first ordering and visual encoder quality matter","More modalities help, but only at the right noise level in ASR","MLLM ASR: visual cues shine when audio gets noisy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic 3-Equations setup—text-to-speech utterances with MUSAN noise on the second half and three displayed equations with two spoken—behaves like real noisy multimodal speech, so the noise-dependent ordering of modality benefits transfers to real-world ASR; the paper also assumes hyperparameters tuned on 3-Equations transfer across model variants, yet notes Mamba needed different settings.","fun_headline_variants_meta":{"raw":{"variants":["Noise decides which extra modality helps MLLM speech recognition","Lip sync aids at high noise, image context at moderate SNR","Multimodal ASR: audio-first ordering and visual encoder quality matter","More modalities help, but only at the right noise level in ASR","MLLM ASR: visual cues shine when audio gets noisy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1392,"prompt_tokens":1015,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":286}},"tokens_in":631,"tokens_out":377,"duration_ms":4118,"temperature":1.0,"reasoning_tokens":286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:02:08.506169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same DMLM audio+OCR and audio+lips models on a real dataset with per-utterance SNR labels and measure relative benefit at SNR values from 20 to -20 dB: if OCR/image benefit does not peak at moderate SNR, or if lip benefit does not rise as SNR falls, the central claim is wrong.","supporting_citations":[{"cited_title":"Multi- modal speech transformer decoders: When do multiple modalities improve accuracy?","cited_arxiv_id":null,"evidence_quote":"Supplies the 3-Equations dataset and the initial DMLM-based modality analysis that this paper extends; the synthetic dataset is the main platform for the noise-dependent claims."},{"cited_title":"Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing","cited_arxiv_id":"2406.06582","evidence_quote":"Defines DMLM, the discrete-token multimodal OPT-125M model and its pretraining and multimodal loss recipe that all experiments inherit."},{"cited_title":"Learning audio-visual speech representation by masked multimodal cluster prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the frozen AV-HuBERT encoder that turns lip movements into discrete tokens, i.e., the synchronized modality at the center of the high-noise claim."},{"cited_title":"Slideavsr: A dataset of paper explanation videos for audio-visual speech recognition,","cited_arxiv_id":null,"evidence_quote":"Provides SlideAVSR, the real-world slide-explanation dataset used to check the unsynchronized OCR/image trend and FQ Ranker filtering."},{"cited_title":"Spoken moments: Learning joint audio-visual representations from video descriptions,","cited_arxiv_id":null,"evidence_quote":"Provides Spoken Moments in Time, the real-world video dataset used for image/video and visual-encoder comparisons."},{"cited_title":"Mamba: Linear-time sequence mod- eling with selective state spaces,","cited_arxiv_id":null,"evidence_quote":"Defines the Mamba architecture used as the alternative backbone in the architecture-comparison experiments."},{"cited_title":"Perceptual score: What data modalities does your model perceive?","cited_arxiv_id":null,"evidence_quote":"Defines the Perceptual Score metric used to attribute WER influence to individual modalities in the 4-modality model."},{"cited_title":"Auto- avsr: Audio-visual speech recognition with automatic labels,","cited_arxiv_id":null,"evidence_quote":"Earlier evidence that synchronized audio-visual benefit grows with noise; the paper's lip-movement trend is an extension of this."}],"review_version":2}