Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a second-pass language model which re-reads the original audio and lip video, together with the AVSR system's N-best hypotheses, lowers word error rate by 24% on LRS3.

desk verdict A genuinely new audio+visual GER combination with good high-SNR results, but the 24% headline overstates it and the WER-loss gradient gap needs a real answer. read the letter →

arxiv 2501.04038 v1 pith:W36XQHUP submitted 2025-01-03 cs.MM cs.AIcs.SDeess.AS

Rui Liu , Hongyu Yuan , Haizhou Li This is my paper · ORCID
classification cs.MMcs.AIcs.SDeess.AS
keywords audio-visualspeechrecognitiongenerativeerrorcorrectionlargelanguagemodelsQ-Formermultimodalsynchronousencodercross-modalpromptingLoRAfine-tuningLRS3benchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that generative error correction for audio-visual speech recognition should not stop at the recognizer's N-best list. AVGER, the proposed system, sends the original audio and lip video through a Q-Former-based encoder a second time, compresses them into tokens an LLM can read, and asks the LLM to pick the best transcription from the N-best hypotheses. On the LRS3 benchmark this reduces word error rate by about 24% relative to the base AVSR system on clean audio, with smaller but consistent gains at every tested noise level. The reason this matters is that it suggests the first-pass recognizer leaves recoverable information in the raw multimodal signal, and that an LLM can exploit it when the audio and visual streams are presented together with the candidates.

What carries the argument

The load-bearing mechanism is the Multimodal Synchronous Encoder built on a shared Q-Former. The Q-Former is a querying transformer that compresses long frame-level features into a fixed number of learnable query vectors; here, the speech and lip-video features are first split into equal-length temporal segments, each segment is compressed separately with modality-specific queries, and the outputs are stacked and linearly bridged into the LLM embedding space. Temporal clipping with non-overlapping segments preserves the synchronization between audio and video, which the authors identify as the key to keeping multimodal alignment. The Cross-modal Prompt then places the audio representation, the video representation, and the N-best candidate list into separate labeled sections of the prompt, and the Multi-Level Consistency Constraint loss ties the audio and video representations to the ground-truth text representation while also optimizing the final output against the reference transcription.

What would settle it

Inspect the released training code for the implementation of the word-error-rate loss: if it has no straight-through estimator, no REINFORCE-style sampling, and no differentiable surrogate, then the objective as written cannot be minimized by gradient descent, and the ablation gains attributed to that loss term are unsupported.

Watch

Extended reading notes

Core claim

The central claim is that an LLM-based second pass can correct AVSR errors if it re-reads the original audio and visual signals rather than working from transcripts alone. AVGER first decodes N-best transcriptions with an AVSR system, then a multimodal synchronous encoder slices the temporally aligned speech and lip-video features into fixed-length segments and compresses each segment with a shared Q-Former using modality-specific query vectors. The compressed audio representation, compressed video representation, and the N-best list are assembled into a cross-modal prompt, and a decoder LLM fine-tuned with LoRA produces the corrected transcription. Training uses a multi-level consistency constraint that combines cross-entropy, a word-error-rate term, and a central-moment discrepancy that pulls the audio and video representations toward the text representation. On LRS3 the method lifts the base AVSR word error rate from 1.45% to 1.10% on clean audio (24.1% relative reduction), outperforms text-only GER and the uncertainty-fusion UADF baseline at clean and high-SNR conditions, and improves over the base system at all tested SNRs.

Load-bearing premise

The training pipeline assumes that the word-error-rate term in the loss, defined as a Levenshtein edit distance and therefore non-differentiable, can still be optimized by backpropagation, but the paper never specifies how its gradients are obtained.

Editorial extensions

If this is right

  • If AVGER's results hold, a second-pass LLM that re-reads the original audio and video can improve AVSR output substantially beyond what the first-pass decoder alone achieves.
  • The clean-audio gain shows visual information is useful even without acoustic noise, not merely as a noise-robustness mechanism.
  • The ablation places the temporal slicing of frame-level features as the largest single contributor, so preserving segment-level synchronization should be a design principle for multimodal speech LLMs.
  • The monotone improvements with more N-best hypotheses and more Q-tokens indicate that candidate diversity and representation capacity directly trade for accuracy in this pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • At -10 dB SNR, the text-only UADF baseline beats AVGER, so a natural extension is a noise-adaptive router that down-weights the audio representation and leans on lip video when the audio is heavily corrupted.
  • Because the non-differentiable word-error-rate loss is added to the total loss without a stated gradient mechanism, some of the reported gain may come from the cross-entropy and CMD terms alone; a controlled run that drops only the WER term would isolate its true contribution.
  • The correction encoder is deliberately independent of the first-pass recognizer, which implies the same second-pass stage could be attached to a different AVSR or even ASR backend without retraining the backend, a transfer the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AVGER, a generative error correction (GER) framework for audio-visual speech recognition (AVSR). A first-stage AVSR system (AV-HuBERT) produces N-best hypotheses; a Q-Former-based multimodal synchronous encoder then re-reads the original audio and lip video, segments the frame-level features into temporally aligned clips, and compresses them into LLM-readable representations. These representations and the N-best list are combined into a cross-modal prompt for a LLaMA decoder fine-tuned with LoRA. Training uses a multi-level loss consisting of cross-entropy, a Levenshtein-distance-based WER term, and a central moment discrepancy (CMD) term. Experiments on LRS3 under clean and babble-noise conditions report WER reductions, with the abstract claiming a 24% WER reduction and superior performance to mainstream AVSR systems.

Significance. The underlying idea is timely and potentially useful: using the original audio and visual signals in a second-pass LLM-based correction stage is a natural extension of recent ASR GER work, and the temporal-segmentation design for modality synchronization is a sensible engineering choice. The paper also provides ablations, hyperparameter studies, and a public code/model link, which are helpful for reproducibility. However, the evidence presented does not support the claims as stated: the headline 24% figure is cherry-picked from the clean/high-SNR rows, low-SNR results are actually worse than the UADF baseline, the training loss as written is not differentiable, and the data-partition description is ambiguous. If these issues are resolved, the contribution could be a useful advance; in its current form the central claims are not established.

major comments (4)
  1. [Results and Analysis, Table 1] The abstract's claim of a 24% WER reduction and "outperforming current mainstream AVSR systems" is not supported by the full table. The 24% figure corresponds to the WERR relative to AV-HuBERT only at 0 dB, 5 dB, and clean conditions; at -10 dB the WERR is 4.0% and at -5 dB it is 8.9%, while UADF achieves a lower WER (21.8 vs. 29.1 at -10 dB; 10.7 vs. 12.3 at -5 dB). The authors should either restrict their claims to the conditions where AVGER is actually superior or provide a substantive analysis of why the proposed method underperforms at low SNR.
  2. [Multi-Level Consistency Constraint Training, Eq. (11) and Eq. (13)] L_WER is defined as a Levenshtein edit distance between the decoded transcription and the ground truth, but no gradient estimator is described. Because edit distance on discrete token sequences is piecewise constant and has zero gradient almost everywhere, the total loss L_MLC is not differentiable as written, so the training procedure is not executable without an additional mechanism such as straight-through estimation, REINFORCE, or a differentiable surrogate. This is load-bearing because the ablation in Table 2 (rows #1 vs. #2 and #1 vs. #3) attributes part of the improvement to L_WER and L_CMD. The authors must specify how gradients are obtained or remove the term from the training objective.
  3. [Dataset] The data-partition description is internally inconsistent. The text says the original training and test sets were combined, re-divided into a 270h training set and a 1h validation set, and that the original 0.9h test set was maintained; these statements cannot all be true simultaneously. If any test clips were used during training, the reported WERs are not valid. The authors must clarify the exact partition and explicitly confirm that the test utterances are disjoint from the training and validation sets.
  4. [Tables 1-3] No error bars, confidence intervals, or multiple-seed results are reported. On a 0.9h test set, differences such as 1.10 vs. 1.19 WER (Table 2, rows #1 and #5) are small and may not be statistically reliable. At least multiple runs or a significance test are needed to support the comparative and ablation claims, especially given that some rows differ by less than 0.1 WER.
minor comments (5)
  1. [Eq. (10)] The CMD loss sums from k=5, whereas the standard CMD definition uses k=2..K. If this is intentional, please explain; otherwise it appears to be a typo that omits variance, skewness, and kurtosis terms.
  2. [Metrics] The text defines WERR as improvement relative to GER, but the values in Table 1 are computed relative to AV-HuBERT (e.g., clean WER 1.45 to 1.10 gives 24.1%). Please make the reference baseline explicit and consistent throughout.
  3. [Throughout] The spelling of AVSR is inconsistent ('AVSR', 'A VSR', 'A VSR'), and there are several typos (e.g., 'rencent', 'Corrposending'). Please standardize the terminology and proofread the manuscript.
  4. [Experimental Setup] The insertion of RepS and RepL into the LLM embedding sequence is described only verbally; please specify whether they are concatenated with text embeddings, how length alignment is handled, and how the model distinguishes audio, video, and text segments.
  5. [Figure 1] The 'Part 1' through 'Part 5' labels in the figure are not explained in the caption, which makes the architecture diagram hard to follow. Please annotate the parts or add a matching description in the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AVGER's reported gains are empirical results from held-out LRS3 test data; no derivation reduces to its own inputs.

full rationale

The paper's central claim is an empirical comparison: AVGER is trained and evaluated on LRS3, with standard baselines, and WER/WERR numbers in Table 1 are measured outputs, not derived from the assumption that they would improve. The N-best hypotheses from AV-HuBERT are inputs, not fitted to the test WER. Equations (9)–(13) define the training loss; even if L_WER in Eq. (11) is non-differentiable as written (a correctness/executability concern), that does not make the evaluation circular because the reported WER is not by construction equal to any training target. No load-bearing conclusion rests on a self-citation: references to HyPoradise, UADF, BLIP-2, and LoRA are independent prior work, and no author self-citation is used to justify the main architecture or to forbid alternatives. Hyperparameter choices (N-best size, Q-token count) are presented as ablations on a validation/test split, not as predictions derived from the same data. The 24% WERR claim is not definitionally true: it is an observed ratio from Table 1 and could have come out otherwise at other SNR levels (indeed WERR is only 4.0% at -10 dB), confirming that the result is not forced by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central empirical claim rests on pretrained models and on a training loss that includes a non-differentiable term as written. Hyperparameters are tuned on a re-split LRS3 validation set.

free parameters (4)
  • N-best hypotheses count = 10
    Tuned on the validation set; WER decreased from 1.17 (N=5) to 1.10 (N=10).
  • Q-Former query tokens = 20
    Tuned on validation; WER decreased from 1.16 (Q=10) to 1.10 (Q=20).
  • Temporal window length = 1 second
    Hand-chosen; no ablation reported for this hyperparameter.
  • LoRA rank = 32
    Hand-chosen; not ablated.
assumptions (4)
  • domain assumption Pretrained LLaMA-7B, HuBERT, VideoMAE, and AV-HuBERT provide useful representations for multimodal error correction.
    The method's success relies on transfer from these pretrained models; no evidence is given that they are sufficient or that the frozen encoders preserve the needed information.
  • domain assumption The N-best hypotheses from AV-HuBERT contain the correct transcription with high probability, so an LLM can select it.
    The error-correction setup is only as good as the candidate list; the paper does not analyze oracle or coverage rates.
  • ad hoc to paper Levenshtein-distance based L_WER can be used as a differentiable training loss.
    Eq. (11) defines L_WER as an edit distance, which is non-differentiable; the paper does not explain how it is optimized (e.g., straight-through estimator, REINFORCE, or approximation).
  • ad hoc to paper Non-overlapping 1-second temporal clipping preserves cross-modal synchronization without losing information.
    The design assumes that fixed 1s windows align audio and video segments; no analysis of alignment error or window length sensitivity is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition." pith.science (2026). https://pith.science/paper/W36XQHUP

@misc{pith2026250104038,
  author       = {Pith},
  title        = {Pith review of: Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W36XQHUP}},
  note         = {Machine review of arXiv:2501.04038}
}
read the original abstract

Unlike traditional Automatic Speech Recognition (ASR), Audio-Visual Speech Recognition (AVSR) takes audio and visual signals simultaneously to infer the transcription. Recent studies have shown that Large Language Models (LLMs) can be effectively used for Generative Error Correction (GER) in ASR by predicting the best transcription from ASR-generated N-best hypotheses. However, these LLMs lack the ability to simultaneously understand audio and visual, making the GER approach challenging to apply in AVSR. In this work, we propose a novel GER paradigm for AVSR, termed AVGER, that follows the concept of ``listening and seeing again''. Specifically, we first use the powerful AVSR system to read the audio and visual signals to get the N-Best hypotheses, and then use the Q-former-based Multimodal Synchronous Encoder to read the audio and visual information again and convert them into an audio and video compression representation respectively that can be understood by LLM. Afterward, the audio-visual compression representation and the N-Best hypothesis together constitute a Cross-modal Prompt to guide the LLM in producing the best transcription. In addition, we also proposed a Multi-Level Consistency Constraint training criterion, including logits-level, utterance-level and representations-level, to improve the correction accuracy while enhancing the interpretability of audio and visual compression representations. The experimental results on the LRS3 dataset show that our method outperforms current mainstream AVSR systems. The proposed AVGER can reduce the Word Error Rate (WER) by 24% compared to them. Code and models can be found at: https://github.com/CircleRedRain/AVGER.

Figures

Figures reproduced from arXiv: 2501.04038 by the authors.

Figure 1
Figure 1. Architecture of the AVGER System. The AVGER system integrates speech and lip video inputs for improved transcrip [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The workflow of Q-Former. ek is a clip of seg￾mented frame-level features, Pk is segment-level position embeddings, and ⊕ denotes the vector addition operation. where e S k ∈ R τS×d and e L k ∈ R τL×d denote the k-th seg￾ment of e S and e L, with K = ⌈T /τ ⌉, τS = TS/K, and τL = TL/K. Segment-level position embeddings {Pk} K k=1, Pk ∈ R d are added to each e S k and e L k . The addition of segment-level position emb… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 26 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    S.; and Zisserman, A

    Afouras, T.; Chung, J. S.; and Zisserman, A. 2018. LRS3-TED: a large-scale dataset for visual speech recognition. CoRR, abs/1809.00496

    arXiv 2018

  4. [4]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; et al. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, N...

    work page 2022

  5. [5]

    H.; Siniscalchi, S

    Chen, C.; Hu, Y.; Yang, C. H.; Siniscalchi, S. M.; Chen, P.; and Siong, C. E. 2023 a . HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information P...

    work page 2023

  6. [6]

    Chen, C.; Hu, Y.; Zhang, Q.; Zou, H.; Zhu, B.; and Chng, E. S. 2023 b . Leveraging Modality-Specific Representations for Audio-Visual Speech Recognition via Reinforcement Learning. In Williams, B.; Chen, Y.; and Neville, J., eds., Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Ar...

    work page 2023

  7. [7]

    M.; Chen, P.; Chng, E.; and Yang, C

    Chen, C.; Li, R.; Hu, Y.; Siniscalchi, S. M.; Chen, P.; Chng, E.; and Yang, C. H. 2024. It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    work page 2024

  8. [8]

    Davis, S.; and Mermelstein, P. 1980. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE transactions on acoustics, speech, and signal processing, 28(4): 357--366

    work page 1980

Show all 40 references
  1. [9]

    Ghosh, S.; Kumar, S.; Seth, A.; Chiniya, P.; Tyagi, U.; Duraiswami, R.; and Manocha, D. 2024. LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition . In Proc. INTERSPEECH 2024

    2024

  2. [10]

    V.; Joulin, A.; and Misra, I

    Girdhar, R.; El - Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. ImageBind One Embedding Space to Bind Them All. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , 15180--15190. IEEE

    2023

  3. [11]

    Gulati, A.; Qin, J.; Chiu, C.; Parmar, N.; Zhang, Y.; Yu, J.; Han, W.; Wang, S.; Zhang, Z.; Wu, Y.; and Pang, R. 2020. Conformer: Convolution-augmented Transformer for Speech Recognition. In Meng, H.; Xu, B.; and Zheng, T. F., eds., 21st Annual Conference of the International ...

    2020

  4. [12]

    H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A

    Hsu, W.-N.; Bolte, B.; Tsai, Y.-H. H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29: 3451--3460

    2021

  5. [13]

    J.; Shen, Y.; Wallis, P.; et al

    Hu, E. J.; Shen, Y.; Wallis, P.; et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net

    2022

  6. [14]

    S.; and Li, R

    Hu, Y.; Chen, C.; Qin, C.; Zhu, Q.; Chng, E. S.; and Li, R. 2024 a . Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models. CoRR, abs/2405.10025

    2024 arXiv

  7. [15]

    H.; Li, R.; Zhang, C.; Chen, P.; and Chng, E

    Hu, Y.; Chen, C.; Yang, C. H.; Li, R.; Zhang, C.; Chen, P.; and Chng, E. 2024 b . Large Language Models are Efficient Learners of Noise-Robust Speech Recognition. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . ...

    2024

  8. [16]

    King, D. E. 2009. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research, 10: 1755--1758

    2009

  9. [17]

    Leng, Y.; Tan, X.; Wang, R.; Zhu, L.; Xu, J.; et al. 2021. FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition. In Moens, M.; Huang, X.; Specia, L.; and Yih, S. W., eds., Findings of the Association for Computational Linguistics: EMNLP ...

    2021

  10. [18]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. C. H. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., International Conference on ...

    2023

  11. [19]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  12. [20]

    Ma, P.; Haliassos, A.; Fernandez - Lopez, A.; Chen, H.; Petridis, S.; and Pantic, M. 2023. Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-1...

    2023

  13. [21]

    Ma, P.; Petridis, S.; and Pantic, M. 2021. End-To-End Audio-Visual Speech Recognition with Conformers. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021 , 7613--7617. IEEE

    2021

  14. [22]

    Maaz, M.; Rasheed, H.; Khan, S.; and Khan, F. S. 2024. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    2024

  15. [23]

    Mikolov, T.; Karafi \' a t, M.; Burget, L.; Cernock \' y , J.; and Khudanpur, S. 2010. Recurrent neural network based language model. In Kobayashi, T.; Hirose, K.; and Nakamura, S., eds., 11th Annual Conference of the International Speech Communication Association, INTERSPEECH...

    2010

  16. [24]

    Mu, B.; Wan, X.; Zheng, N.; Zhou, H.; and Xie, L. 2024. MMGER: Multi-Modal and Multi-Granularity Generative Error Correction With LLM for Joint Accent and Speech Recognition. IEEE Signal Processing Letters, 31: 1940--1944

    2024

  17. [25]

    OpenAI. 2023. GPT-4 Technical Report. CoRR, abs/2303.08774

    2023 arXiv

  18. [26]

    W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I

    Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., International Conference on Machine Learn...

    2023

  19. [27]

    H.; Khan, S

    Radhakrishnan, S.; Yang, C. H.; Khan, S. A.; Kumar, R.; Kiani, N. A.; Gomez - Cabrero, D.; and Tegn \' e r, J. 2023. Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2...

    2023

  20. [28]

    Sennrich, R.; Haddow, B.; and Birch, A. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers . The Asso...

    2016

  21. [29]

    Shi, B.; Hsu, W.; Lakhotia, K.; and Mohamed, A. 2022. Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net

    2022

  22. [30]

    Shi, B.; Hsu, W.; and Mohamed, A. 2022. Robust Self-Supervised Audio-Visual Speech Recognition. In Ko, H.; and Hansen, J. H. L., eds., 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, 2118--2...

    2022

  23. [31]

    Snyder, D.; Chen, G.; and Povey, D. 2015. MUSAN: A Music, Speech, and Noise Corpus. CoRR, abs/1510.08484

    2015 arXiv

  24. [32]

    Tang, C.; Yu, W.; Sun, G.; Chen, X.; Tan, T.; Li, W.; Lu, L.; Ma, Z.; and Zhang, C. 2024. SALMONN: Towards Generic Hearing Abilities for Large Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . Ope...

    2024

  25. [33]

    Tong, Z.; Song, Y.; Wang, J.; and Wang, L. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 10078--10093

    2022

  26. [34]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; et al. 2023 a . LLaMA: Open and Efficient Foundation Language Models. CoRR, abs/2302.13971

    2023 arXiv

  27. [35]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; et al. 2023 b . Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR, abs/2307.09288

    2023 arXiv

  28. [36]

    N.; Kaiser, L.; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is All you Need. In Guyon, I.; von Luxburg, U.; Bengio, S.; Wallach, H. M.; Fergus, R.; Vishwanathan, S. V. N.; and Garnett, R., eds., Advances in Neura...

    2017

  29. [37]

    Xu, Y.; Chen, H.; Yu, J.; Huang, Q.; Wu, Z.; et al. 2024. SECap: Speech Emotion Captioning with Large Language Model. In Wooldridge, M. J.; Dy, J. G.; and Natarajan, S., eds., Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innov...

    2024

  30. [38]

    Yu, W.; Tang, C.; Sun, G.; Chen, X.; Tan, T.; Li, W.; Lu, L.; Ma, Z.; and Zhang, C. 2024. Connecting Speech Encoder and Large Language Model for ASR . In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-1...

    2024

  31. [39]

    Zellinger, W.; Grubinger, T.; Lughofer, E.; Natschl \" a ger, T.; and Saminger - Platz, S. 2017. Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, ...

    2017

  32. [40]

    Zhang, H.; Li, X.; and Bing, L. 2023. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. In Feng, Y.; and Lefever, E., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - System Demonstr...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.