Pith. sign in

REVIEW 4 major objections 5 minor 58 references

MLLM-based Speech Recognition: When and How is Multimodality Beneficial?

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that in multimodal LLM speech recognition, synchronized modalities like lip movements help most as noise rises, while unsynchronized image/OCR context helps most at moderate noise.

desk verdict A solid extended ablation of multimodal ASR, but the abstract overclaims two findings that the paper's own tables undercut, and the 3-Equations noise schedule confounds the headline 'when' result. read the letter →

arxiv 2507.19037 v1 pith:OARVQ6EP submitted 2025-07-25 cs.SD cs.CLcs.MMeess.AS

Yiwen Guan , Viet Anh Trinh , Vivek Voleti , Jacob Whitehill This is my paper · ORCID
classification cs.SDcs.CLcs.MMeess.AS
keywords multimodallargelanguagemodelsautomaticspeechrecognitionaudio-visualnoiserobustnesslipmovementsvisualcontextMambastatespacemodalityfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that multimodality is not uniformly beneficial for LLM-based speech recognition: the value of each extra input channel is governed by how well it is synchronized with the audio and by the acoustic noise level. Specifically, lip movements, which track the speech signal in time, become more useful as SNR decreases, whereas static image or OCR context is most useful when noise is moderate—strong enough that audio alone is unreliable, weak enough that the model can still align the speech to the context. The authors support this with a discrete-token multimodal language model (DMLM) on a synthetic equations-reading dataset and two real-world video datasets, and they additionally report that better visual representations and encoders improve ASR, Mamba backbones follow the same qualitative trends, and input order and loss weights matter. If correct, the work turns 'add more modalities' into a design rule: choose modalities by noise regime, not by count.

What carries the argument

The load-bearing object is DMLM, a discrete-token multimodal decoder-only language model: frozen modality encoders convert audio, lips, and images into token sequences that are concatenated and processed by an OPT-125M backbone trained with a modality-weighted cross-entropy loss. The paper's main analytical instruments are the relative benefit (RB) and perceptual score (PS) metrics, which quantify how much a modality adds or accounts for in WER terms. Two design manipulations carry the argument: varying SNR with MUSAN noise so that synchronized versus unsynchronized alignment can be compared, and moving visual input along a quality continuum from raw image to OCR to oracle-OCR.

What would settle it

Run the same DMLM audio+OCR and audio+lips models on a real dataset with per-utterance SNR labels and measure relative benefit at SNR values from 20 to -20 dB: if OCR/image benefit does not peak at moderate SNR, or if lip benefit does not rise as SNR falls, the central claim is wrong.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is a noise-dependent ordering of visual modality benefit in MLLM-based ASR. Using DMLM—a frozen-encoder decoder-only OPT-125M that tokenizes speech via Seamless, lips via AV-HuBERT, and images via DALL-E or ViT—the authors vary the input modalities on the 3-Equations synthetic corpus and measure relative benefit RB=(WER(A)−WER(A∪S))/WER(A). They find that adding OCR context to audio yields its highest RB at moderate SNR (about 33.5% at 2.5 dB, versus 12% at 10 dB and 9% at -20 dB), while adding synchronized lip movements yields an RB that is negative or near zero at high SNR and grows to about +10% at -20 dB. They also find that adding a fourth modality is not always better at low noise, that higher-quality visual representations (oracle OCR vs raw image) and stronger visual encoders improve accuracy, that Mamba-130M reproduces the OCR/lip trends but with lower accuracy and more hyperparameter sensitivity, and that audio-first input ordering and nonzero modality loss weights help. Similar unsynchronized-context trends appear on SlideAVSR with slide and OCR inputs.

Load-bearing premise

The load-bearing premise is that the synthetic 3-Equations setup—text-to-speech utterances with MUSAN noise on the second half and three displayed equations with two spoken—behaves like real noisy multimodal speech, so the noise-dependent ordering of modality benefits transfers to real-world ASR; the paper also assumes hyperparameters tuned on 3-Equations transfer across model variants, yet notes Mamba needed different settings.

Editorial extensions

If this is right

  • Adding more modalities to an MLLM ASR model is not universally helpful: the benefit is positive on average, but at low noise the 4-modality model underperforms 3-modality ones, so context length should be treated as a cost.
  • For unsynchronized context such as slides, images, or OCR, deployment is most valuable at moderate noise; at very low SNR the context no longer aligns with the speech and can hurt accuracy.
  • For synchronized lip input, the value grows with noise, so audio-visual front-ends are most justified in high-noise settings.
  • Improving visual encoders, especially their semantic quality, pays off directly in ASR accuracy even when the decoder is unchanged.
  • Mamba can reproduce the qualitative modality trends while training much faster, but its sensitivity to hyperparameters means tuned settings from a Transformer model should not be assumed to transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication left implicit by the paper is a noise-adaptive inference policy: estimate per-utterance SNR, then route to image/OCR context at moderate noise and to lip input at high noise; the paper measures benefit per SNR bucket but does not build such a controller.
  • Since the authors attribute the 4-modality model's failure partly to sequence noise, I infer that compressing or filtering context tokens before concatenation should restore much of the multimodal gain at low SNR; a direct test would compare OCR with token pruning at SNR -10 dB.
  • The ViT-versus-DALL-E gap suggests semantic visual encoders aligned with language are more valuable for MLLM ASR than structural encoders; I would test this by scaling each encoder while holding token count fixed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript presents a systematic empirical study of when and how supplementary visual and lip modalities improve MLLM-based ASR under acoustic noise. Using a synthetic 3-Equations dataset and two real-world corpora (SlideAVSR, S-MiT), it evaluates OPT- and Mamba-based DMLM models, computes WER, relative benefit (RB), and perceptual scores (PS) across SNR levels, and additionally studies visual representation quality, sequence noise, modality input order, and modality-weighted loss components. The central claims are that adding modalities usually helps ASR, synchronized modalities (lip movements) help most at high noise while unsynchronized modalities (image/OCR) peak at moderate noise, better visual representations help, Mamba exhibits similar multimodality trends to Transformers, and input format and loss weights significantly affect accuracy.

Significance. The study's controlled manipulation of SNR and the reported statistical window around minibatch shuffling are genuine strengths; the paper does not rely on a single arbitrary training run, and the comparisons across modality combinations are unusually systematic for this area. If the claims were fully supported, the paper would provide practical guidance for designing multimodal ASR front-ends and for choosing backbones, input encodings, and loss balancing. However, several of the abstract's global conclusions are not consistently supported by the paper's own tables, and the main noise-dependent 'when' result is derived from a synthetic corpus with a positional noise confound. The contribution nevertheless contains useful observations (e.g., sequence-noise effects, OCR benefits, negative results for raster and interleaved inputs) that deserve publication after substantial revision.

major comments (4)
  1. [Sections IV-A and VII-B] The central 'when' finding (Fig. 3 and Table I) is confounded by the 3-Equations noise schedule: MUSAN noise is added only to the second half of each utterance, and Section VII-B demonstrates that, with absolute positional encodings, the model can exploit the known location of the clean first half and that this location interacts with modality order. Because the synchronized-lip trend (increasing RB as SNR decreases) is the only controlled evidence for the abstract's claim (2), this trend may reflect the position of the clean half rather than the acoustic SNR itself. Please add a control experiment with stationary full-utterance noise on 3-Equations, or explicitly restrict the synchronized-modality claim to partial-noise conditions.
  2. [Section V-D, Table I; abstract claim (3)] The abstract's claim that 'higher-quality visual representations consistently improve ASR accuracy' is contradicted by the raster row of Table I: R+A has negative RB at every SNR (average -11.7%) despite raster being introduced as a 'perfect' 2D representation, and it is worse than both image and OCR. The explanation about sequence length and 2D positional encodings is plausible, but the blanket claim must be qualified to 'more abstract and informative representations' or to representations for which the model has adequate positional support, and the abstract should be revised accordingly.
  3. [Section VI, Table IV; abstract claim (4)] The abstract's claim that 'Mamba exhibits similar trends regarding the benefits of multimodality as do Transformers' is not supported by the reported numbers. On Mamba, I+A RB decreases as noise increases (from +10.8% at 20 dB to -2.4% at -20 dB), which the paper itself in Section VI says is 'different from the trend on OPT,' and L+A RB is negative at every SNR on Mamba. Moreover, Section VI reports that Mamba required different hyperparameters for different modality combinations, so using the 3-Equations-tuned lambda values for all models may unfairly penalize some Mamba configurations. The claim should be restricted to OCR-only or supported by per-combination hyperparameter tuning for each reported configuration.
  4. [Section V-B and Fig. 4; abstract claim (1)] The abstract's causal framing that 'each modality provides complementary information' is in tension with the paper's own Perceptual Score analysis: Section V-B states that PS(I) is near zero for most SNR values and that the image's benefit 'might have more to do with regularization than complementary information.' If the image modality contributes mainly through regularization, the claim about complementary information should be reformulated, and the paper should provide a test that separates complementary-information benefits from regularization effects (e.g., an example where the image resolves an ambiguity that other modalities cannot resolve).
minor comments (5)
  1. [Section VI] The training-time sentence is garbled: 'for an I+A → T model, the training time of OPT grows to 305 sec/ep (55% increase), while OPT it increased to 643 sec/ep (68% increase)' appears to repeat 'OPT' where 'Mamba' is intended; please clarify which architecture corresponds to each number.
  2. [Section III-C] The grid search over lambda_S, lambda_T, and lambda_I is not specified; please report the candidate values and the validation criterion used so that the tuning procedure is reproducible.
  3. [Section V-A] The sentence 'we find evidence that multimodal ASR can benefit from at least 4 modalities as inputs' is ambiguous; it should be reworded to refer to the 4-modality model or to the number of added modalities.
  4. [Section VII-A] The 'blocked' input format is defined only by contrast with the interleaved format; please state explicitly whether 'blocked' means all audio tokens followed by all frame tokens (or vice versa) and whether the five video-frame token sequences are contiguous.
  5. [General] The manuscript repeatedly refers to Supplementary Materials Sections I-VI for details that support main-text claims (e.g., the exact minibatch shuffling procedure); please ensure the supplementary material is available to reviewers and that the main text summarizes the key procedural details.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical measurements, not consequences of definitions or self-citations.

full rationale

The paper's conclusions are empirical results obtained by training models and measuring WER/RB across SNR levels, not derivations from definitions. Equation (2) defines RB but does not force any particular noisy-SNR trend; the observed 'sweet spot' for unsynchronized modalities and increasing lip benefit comes from the experiments in Fig. 3 and Table I. The 3-Equations dataset is deliberately constructed (MUSAN noise on the second half; two of three equations spoken), which shapes the task, but the models' behavior is still measured rather than logically entailed, and Section VII-B candidly identifies a positional-encoding confound in the audio-order analysis — a validity/generalizability caveat, not circular reasoning. Self-citations [1] and [5] supply the DMLM architecture and the 3-Equations dataset, but they are not used to prove the new claims; the Mamba comparison, input-format experiments, and visual-encoder comparisons are new empirical evidence, and SlideAVSR provides independent real-world support for the unsynchronized-modality sweet spot. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no equation reduces to its own input by construction. Therefore the paper is self-contained in its empirical derivation chain.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The paper's conclusions rest on the representativeness of one synthetic dataset, a specific small-model architecture, and hyperparameters tuned on that same dataset. No new theoretical entities are introduced.

free parameters (1)
  • modality loss weights (λ_S, λ_T, λ_I) = (0.3, 0.5, 0.5)
    Grid-searched on a validation subset of 3-Equations to minimize WER (Section III-C); these weights are then used in training the compared models, potentially favoring the configurations they were tuned for.
assumptions (6)
  • domain assumption 3-Equations is representative of real-world noisy multimodal speech recognition
    Central noise-dependence findings are drawn mostly from this synthetic dataset (Section IV-A).
  • domain assumption MUSAN noise applied to the second half of utterances models auditory noise
    Section IV-A and V: noise is added in a structured way, which may not match real noise patterns.
  • domain assumption Visual representation continuum from image to OCR to raster to oracle-OCR is increasing in quality
    Section IV-A: the authors assert this ordering, but raster underperforms, so the ordering is not consistently reflected in results.
  • domain assumption DMLM with OPT-125M / Mamba-130M is a representative MLLM-based ASR system
    Section III-B: all experiments use this one architecture family, with models much smaller than production MLLMs.
  • domain assumption Absolute positional encodings are optimal for these multimodal inputs
    Section III-B: they tried other strategies (Supplementary II) and found absolute best; this choice shapes input-order effects.
  • domain assumption The modality loss weights tuned on 3-Equations transfer to other datasets and model variants
    Section III-C: weights were optimized on 3-Equations validation; Section VI notes Mamba needs different hyperparameters, so transfer is questionable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLLM-based Speech Recognition: When and How is Multimodality Beneficial?." pith.science (2026). https://pith.science/paper/OARVQ6EP

@misc{pith2026250719037,
  author       = {Pith},
  title        = {Pith review of: MLLM-based Speech Recognition: When and How is Multimodality Beneficial?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OARVQ6EP}},
  note         = {Machine review of arXiv:2507.19037}
}
read the original abstract

Recent advances in multi-modal large language models (MLLMs) have opened new possibilities for unified modeling of speech, text, images, and other modalities. Building on our prior work, this paper examines the conditions and model architectures under which multiple input modalities can improve automatic speech recognition (ASR) accuracy in noisy environments. Through experiments on synthetic and real-world data, we find that (1) harnessing more modalities usually improves ASR accuracy, as each modality provides complementary information, but the improvement depends on the amount of auditory noise. (2) Synchronized modalities (e.g., lip movements) are more useful at high noise levels whereas unsynchronized modalities (e.g., image context) are most helpful at moderate noise levels. (3) Higher-quality visual representations consistently improve ASR accuracy, highlighting the importance of developing more powerful visual encoders. (4) Mamba exhibits similar trends regarding the benefits of multimodality as do Transformers. (5) The input order of modalities as well as their weights in the loss function can significantly impact accuracy. These findings both offer practical insights and help to deepen our understanding of multi-modal speech recognition under challenging conditions.

Figures

Figures reproduced from arXiv: 2507.19037 by the authors.

Figure 1
Figure 1. An overview of Discrete Multimodal Language Model (D [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Datasets used in our work. From left to right: Example [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. RB (%) of adding image, OCR, or lip modalities on 3-Equ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Perceptual scores based on WER of each modality in mod [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: RB of audio-first or audio-later inputs on OPT and Mamb [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 43 canonical work pages

  1. [1]

    Multi- modal speech transformer decoders: When do multiple modalities improve accuracy?

    Y . Guan, V . A. Trinh, V . V oleti, and J. Whitehill, “Multi- modal speech transformer decoders: When do multiple modalities improve accuracy?” arXiv:2409.09221, 2024

    work page arXiv 2024

  2. [2]

    Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann et al. , “Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” arXiv:2507.06261, 2025

    arXiv 2025

  3. [3]

    Next-gpt: Any-to-any multimodal llm,

    S. Wu, H. Fei, L. Qu et al. , “Next-gpt: Any-to-any multimodal llm,” in F orty-first ICML, 2024

    work page 2024

  4. [4]

    Multimodal mamba: Decoder-only multimodal state space model via quadratic to linear distillation,

    B. Liao, H. Tao, Q. Zhang et al. , “Multimodal mamba: Decoder-only multimodal state space model via quadratic to linear distillation,” arXiv:2502.13145, 2025

    arXiv 2025

  5. [5]

    Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing

    V . A. Trinh, R. Southwell, Y . Guan et al. , “Discrete multimodal transformers with a pretrained large lan- guage model for mixed-supervision speech processing,” arXiv:2406.06582, 2024

    work page Pith review arXiv 2024

  6. [6]

    Exploring speech recognition, translation, and understanding with discret e speech units: A comparative study,

    X. Chang, B. Y an, K. Choi et al. , “Exploring speech recognition, translation, and understanding with discret e speech units: A comparative study,” in ICASSP, 2024

    work page 2024

  7. [7]

    Speech recognition meets large language model: Benchmarking, models, and exploration,

    Z. Ma, G. Y ang, Y . Y ang et al. , “Speech recognition meets large language model: Benchmarking, models, and exploration,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2025

    work page 2025

  8. [8]

    Hubert: Self- supervised speech representation learning by masked prediction of hidden units,

    W .-N. Hsu, B. Bolte, Y .-H. H. Tsai et al. , “Hubert: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing , 2021

    work page 2021

Show all 58 references
  1. [9]

    A comprehensive review of multimodal large language models: Performance and challenges across different tasks,

    J. Wang, H. Jiang, Y . Liu et al., “A comprehensive review of multimodal large language models: Performance and challenges across different tasks,” CoRR, 2024

    2024

  2. [10]

    X-llm: Bootstrap- ping advanced large language models by treating multi- modalities as foreign languages,

    F. Chen, M. Han, H. Zhao et al. , “X-llm: Bootstrap- ping advanced large language models by treating multi- modalities as foreign languages,” arXiv:2305.04160, 2023

    2023 arXiv

  3. [11]

    Mms-llama: Efficient llm-based audio-visual speech recognition with minimal multimodal speech tokens,

    J. H. Y eo, H. Rha, S. J. Park, and Y . M. Ro, “Mms-llama: Efficient llm-based audio-visual speech recognition with minimal multimodal speech tokens,” arXiv:2503.11315, 2025

    2025 arXiv

  4. [12]

    Large lan- guage models are strong audio-visual speech recognition learners,

    U. Cappellazzo, M. Kim, H. Chen et al. , “Large lan- guage models are strong audio-visual speech recognition learners,” in ICASSP, 2025

    2025

  5. [13]

    Watch or listen: Robust audio-visual speech recognition with visua l corruption modeling and reliability scoring,

    J. Hong, M. Kim, J. Choi, and Y . M. Ro, “Watch or listen: Robust audio-visual speech recognition with visua l corruption modeling and reliability scoring,” in CVPR, 2023

    2023

  6. [14]

    Large lan- guage models are efficient learners of noise-robust speech recognition,

    Y . Hu, C. Chen, C.-h. H. Y ang et al. , “Large lan- guage models are efficient learners of noise-robust speech recognition,” in ICLR, 2024

    2024

  7. [15]

    Avatar: Un- constrained audiovisual speech recognition,

    V . Gabeur, P . H. Seo, A. Nagrani et al. , “Avatar: Un- constrained audiovisual speech recognition,” in INTER- SPEECH, 2022

    2022

  8. [16]

    Mmger: Multi-modal and multi-granularity generative error correction with ll m for joint accent and speech recognition,

    B. Mu, X. Wan, N. Zheng et al. , “Mmger: Multi-modal and multi-granularity generative error correction with ll m for joint accent and speech recognition,” IEEE Signal Processing Letters, 2024

    2024

  9. [17]

    Mamba: Linear-time sequence mod- eling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence mod- eling with selective state spaces,” in First Conference on Language Modeling , 2023

    2023

  10. [18]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in CVPR, 2024

    2024

  11. [19]

    Qwen-audio: Advanc- ing universal audio understanding via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou et al. , “Qwen-audio: Advanc- ing universal audio understanding via unified large-scale audio-language models,” arXiv:2311.07919, 2023

    2023 arXiv

  12. [20]

    Mm-interleaved: Interleaved image-text generative modeling via multi- modal feature synchronizer,

    C. Tian, X. Zhu, Y . Xiong et al. , “Mm-interleaved: Interleaved image-text generative modeling via multi- modal feature synchronizer,” CoRR, 2024

    2024

  13. [21]

    Transformers are ssms: Generalized IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, AUGUST 2021 10 models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are ssms: Generalized IEEE TRANSACTIONS ON MULTIMEDIA, VOL. XXX, AUGUST 2021 10 models and efficient algorithms through structured state space duality,” in ICML, 2024

    2021

  14. [22]

    Cobra: Extending mamba to multi-modal large language model for efficient inference,

    H. Zhao, M. Zhang, W . Zhao et al. , “Cobra: Extending mamba to multi-modal large language model for efficient inference,” in Proceedings of the AAAI Conference , 2025

    2025

  15. [23]

    U-mamba: Enhancing long- range dependency for biomedical image segmentation,

    J. Ma, F. Li, and B. Wang, “U-mamba: Enhancing long- range dependency for biomedical image segmentation,” arXiv:2401.04722, 2024

    2024 arXiv

  16. [24]

    Vl-mamba: Exploring state space models for multimodal learning,

    Y . Qiao, Z. Y u, L. Guoet al., “Vl-mamba: Exploring state space models for multimodal learning,” CoRR, 2024

    2024

  17. [25]

    Foundations & trends in multimodal machine learning: Principles, chal- lenges, and open questions,

    P . P . Liang, A. Zadeh, and L.-P . Morency, “Foundations & trends in multimodal machine learning: Principles, chal- lenges, and open questions,” ACM Computing Surveys , 2024

    2024

  18. [26]

    Diffusion- lm improves controllable text generation,

    X. Li, J. Thickstun, I. Gulrajani et al. , “Diffusion- lm improves controllable text generation,” Advances in neural information processing systems , 2022

    2022

  19. [27]

    Large language diffusion models,

    S. Nie, F. Zhu, Z. Y ou et al. , “Large language diffusion models,” in ICLR 2025 W orkshop on Deep Generative Model in Machine Learning

    2025

  20. [28]

    Speechgpt: Empow- ering large language models with intrinsic cross-modal conversational abilities,

    D. Zhang, S. Li, X. Zhang et al. , “Speechgpt: Empow- ering large language models with intrinsic cross-modal conversational abilities,” in Findings of the Association for Computational Linguistics: EMNLP , 2023

    2023

  21. [29]

    Mamba in speech: Towards an alternative to self-attention,

    X. Zhang, Q. Zhang, H. Liu et al. , “Mamba in speech: Towards an alternative to self-attention,” IEEE Transac- tions on Audio, Speech and Language Processing , 2025

    2025

  22. [30]

    Where visual speech meets language: Vsp-llm framework for efficient and context-aware visual speech processing,

    J. H. Y eo, S. Han, M. Kim, and Y . M. Ro, “Where visual speech meets language: Vsp-llm framework for efficient and context-aware visual speech processing,” in EMNLP. Association for Computational Linguistics, 2024

    2024

  23. [31]

    Audiopalm: A large language model that can speak and listen,

    P . K. Rubenstein, C. Asawaroengchai, D. D. Nguyen et al. , “Audiopalm: A large language model that can speak and listen,” arXiv:2306.12925, 2021

    2021 arXiv

  24. [32]

    Avformer: Injecting vision into frozen speech models for zero-shot av-asr,

    P . H. Seo, A. Nagrani, and C. Schmid, “Avformer: Injecting vision into frozen speech models for zero-shot av-asr,” in CVPR, 2023

    2023

  25. [33]

    Gesture-aware zero-shot speech recognition for patients with language disorders,

    S. Kim, D. Lee, B. Stark, and J. Han, “Gesture-aware zero-shot speech recognition for patients with language disorders,” in W orkshop on Large Language Models and Generative AI for Health at AAAI 2025 , 2025

    2025

  26. [34]

    Zero-avsr: Zero-shot audio-visual speech recognition with llms by learning language-agnostic speech representations,

    J. H. Y eo, M. Kim, C. W . Kim et al. , “Zero-avsr: Zero-shot audio-visual speech recognition with llms by learning language-agnostic speech representations,” arXiv:2503.06273, 2025

    2025 arXiv

  27. [35]

    Adaptive audio-visual speech recognition via matryoshka-based multimodal llms,

    U. Cappellazzo, M. Kim, and S. Petridis, “Adaptive audio-visual speech recognition via matryoshka-based multimodal llms,” arXiv:2503.06362, 2025

    2025 arXiv

  28. [36]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W . Kim, T. Xu et al. , “Robust speech recognition via large-scale weak supervision,” in ICML, 2023

    2023

  29. [37]

    Perceptual score: What data modalities does your model perceive?

    I. Gat, I. Schwartz, and A. Schwing, “Perceptual score: What data modalities does your model perceive?” Ad- vances in Neural Information Processing Systems , 2021

    2021

  30. [38]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal et al. , “Opt: Open pre-trained transformer language models,” arXiv:2205.01068, 2022

    2022 arXiv

  31. [39]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015

    2015

  32. [40]

    Cvss corpus and massively multilingual speech-to- speech translation,

    Y . Jia, M. T. Ramanovich, Q. Wang, and H. Zen, “Cvss corpus and massively multilingual speech-to- speech translation,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference , 2022

    2022

  33. [41]

    Covost 2 and massively multilingual speech-to-text translation,

    C. Wang, A. Wu, and J. Pino, “Covost 2 and massively multilingual speech-to-text translation,” arXiv:2007.10310, 2020

    2007 arXiv

  34. [42]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie et al., “Microsoft coco: Common objects in context,” in Computer vision–ECCV . Springer, 2014

    2014

  35. [43]

    Seam- lessm4t: Massively multilingual & multimodal machine translation,

    L. Barrault, Y .-A. Chung, M. C. Meglioli et al. , “Seam- lessm4t: Massively multilingual & multimodal machine translation,” arXiv:2308.11596, 2023

    2023 arXiv

  36. [44]

    Learning audio-visual speech representation by masked multimodal cluster prediction,

    B. Shi, W .-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” in ICLR, 2022

    2022

  37. [45]

    Zero-shot text- to-image generation,

    A. Ramesh, M. Pavlov, G. Goh et al. , “Zero-shot text- to-image generation,” in ICML, 2021

    2021

  38. [46]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” in ICLR, 2020

    2020

  39. [47]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR, 2017

    2017

  40. [48]

    Deep audio- visual speech recognition,

    T. Afouras, J. S. Chung, A. Senior et al. , “Deep audio- visual speech recognition,” IEEE transactions on pattern analysis and machine intelligence , 2018

    2018

  41. [49]

    Slideavsr: A dataset of paper explanation videos for audio-visual speech recognition,

    H. Wang, S. Kurita, S. Shimizu, and D. Kawahara, “Slideavsr: A dataset of paper explanation videos for audio-visual speech recognition,” in W orkshop on Ad- vances in Language and Vision Research , 2024

    2024

  42. [50]

    Spoken moments: Learning joint audio-visual representations from video descriptions,

    M. Monfort, S. Jin, A. Liu et al. , “Spoken moments: Learning joint audio-visual representations from video descriptions,” in CVPR, 2021

    2021

  43. [51]

    pyttsx3: Offline text to speech (tts) converter for python,

    “pyttsx3: Offline text to speech (tts) converter for python,” https://github.com/nateshmbhat/pyttsx3

  44. [52]

    Musan: A music, speech, and noise corpus,

    D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” arXiv:1510.08484, 2015

    2015 arXiv

  45. [53]

    Easyocr: Ready-to-use ocr with 80+ supported lan- guages,

    “Easyocr: Ready-to-use ocr with 80+ supported lan- guages,” https://github.com/JaidedAI/EasyOCR

  46. [54]

    Multi- moments in time: Learning and interpreting models for multi-action video understanding,

    M. Monfort, B. Pan, K. Ramakrishnan et al. , “Multi- moments in time: Learning and interpreting models for multi-action video understanding,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021

    2021

  47. [55]

    Auto- avsr: Audio-visual speech recognition with automatic labels,

    P . Ma, A. Haliassos, A. Fernandez-Lopez et al. , “Auto- avsr: Audio-visual speech recognition with automatic labels,” in ICASSP, 2023

    2023

  48. [56]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P . Luc et al. , “Flamingo: a visual language model for few-shot learning,” Advances in neural information processing systems , 2022

    2022

  49. [57]

    Order matters: Exploring order sensitivity in multimodal large language models,

    Z. Tan, X. Chu, W . Li, and T. Mo, “Order matters: Exploring order sensitivity in multimodal large language models,” arXiv:2410.16983, 2024

    2024 arXiv

  50. [58]

    Image first or text first? optimising the sequencing of modalities in large language model prompting and reasoning tasks,

    G. Wardle and T. Suˇ snjak, “Image first or text first? optimising the sequencing of modalities in large language model prompting and reasoning tasks,” Big Data and Cognitive Computing , 2025

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.