Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions

Carlos-D. Mart\'inez-Hinarejos; David Gimeno-G\'omez

REVIEW 2 major objections 6 minor 72 references

Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions

T0 review · 2 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A visual-only system reaches 24.8% WER on Spanish studio lipreading and 34.5% on TV newscasts.

desk verdict Solid empirical step for Spanish lipreading: fine-tuning an existing Conformer gives huge WER gains and a new four-corpus benchmark, with honest limitations; single-seed runs are a caveat, not a fatal flaw. read the letter →

arxiv 2502.00464 v2 pith:JZYNZHZT submitted 2025-02-01 cs.CV

David Gimeno-Gómez , Carlos-D. Martínez-Hinarejos This is my paper · ORCID

classification cs.CV

keywords VisualSpeechRecognitionlipreadingSpanishCTC/AttentionConformerbenchmarkworderrorrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that an end-to-end visual speech recognition system, built on the hybrid CTC/Attention architecture and fine-tuned from English-pretrained weights, brings continuous Spanish lipreading from a feasibility exercise to usable accuracy. On the controlled VLRF corpus the best configuration reaches 24.8% word error rate, against a prior best of 59.7%; on LIP-RTVE television newscasts it reaches 34.5% in the speaker-dependent setting and 59.5% in the speaker-independent setting, where previous systems sat near 80% and 95%. The authors also consolidate a four-corpus Spanish lipreading benchmark spanning studio recordings, TV newscasts, YouTube vlogs, and TED talks, with ablations showing which components carry the accuracy. If the results hold, silent, visual-only Spanish speech recognition is no longer limited to laboratory conditions.

What carries the argument

The load-bearing mechanism is the hybrid CTC/Attention training and inference scheme, in which a CTC branch imposes speech-like local alignment while an attention decoder handles global text conditioning, trained jointly and combined with a character-level Transformer language model during beam search. Around it sits a visual front end that replaces the first layer of a ResNet with a 3D convolution to capture lip motion across five frames, feeding a twelve-layer Conformer encoder that mixes local convolutional and global attention features. The paper further depends on transfer: pretrained English visual speech weights and a general Spanish language model are fine-tuned on each target corpus, and the no-fine-tuning rows in the ablation show that without this initialization the system collapses on VLRF.

What would settle it

Retrain the VLRF and LIP-RTVE configurations multiple times with fresh random seeds and from random initialization without English pretraining. If the seed-to-seed spread in WER approaches the claimed margins over the previous state of the art, or if removing the English pretraining keeps error rates above the old baselines, the conclusion that the architecture itself is responsible for the gains would be weakened.

Watch

Extended reading notes

Core claim

The central claim is that a single end-to-end architecture, consisting of a 3D-convolution front end, a Conformer encoder, a CTC/Attention decoder, and a character-level Transformer language model integrated by shallow fusion, can be fine-tuned to outperform every previously published continuous Spanish lipreading system when it is initialized from English-pretrained visual speech weights. The paper reports 24.8% WER on VLRF, 34.5% on the speaker-dependent LIP-RTVE partition, and 59.5% on the speaker-independent partition, with the ablation study attributing the largest degradation to removing the CTC branch and the language model. The authors present this as evidence that the architecture adapts to different data conditions, and they add CMU-MOSEAS and MuAViC Spanish results from the pretrained model to form a four-way benchmark for future work.

Load-bearing premise

The headline gains rest on the assumption that one training run per configuration is representative, and that the English-pretrained front end transfers to Spanish lips; the paper reports no repeated seeds, and its own no-fine-tuning rows show the pretrained weights are doing most of the work.

Editorial extensions

If this is right

Visual-only Spanish lipreading is workable in controlled and broadcast settings, with the best reported WERs of 24.8% on VLRF and 34.5% on speaker-dependent LIP-RTVE.
The CTC branch is the most important decoder component: dropping it raises WER by 15.9 to 35.6 points, while dropping the attention decoder raises it by 9.1 to 18.6 points.
The language model matters substantially: removing it costs 5.3 to 11.1 points, and fine-tuning it to the domain pays off most on TV newscasts.
The proposed four-corpus benchmark gives later Spanish lipreading work fixed points of comparison across controlled recordings, TV newscasts, YouTube vlogs, and TED talks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same fine-tuning recipe may transfer to other low-resource languages, but the authors do not test this; the large no-fine-tuning gap suggests the English-pretrained visual encoder is supplying most of the skill.
The Zipf and vocabulary-coverage analysis points to a cheap data-selection heuristic for future corpora: prioritize utterances whose words fall in the high-frequency head of the target domain, since test coverage by the top 1000 words tracks task difficulty.
Because each configuration was run once, the ablation order of importance is provisional; a repeated-seed study could plausibly reorder the CTC and language-model contributions.
In practical terms, 24.8% and 34.5% WER make visual-only Spanish lipreading usable for constrained tasks such as keyword spotting or studio dictation, while 59.5% on open-domain speaker-independent newscasts remains too error-prone for unconstrained use.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Ping. This paper is a solid empirical contribution to Spanish VSR. The headline result is real: fine-tuning Ma et al.'s CTC/Attention Conformer on VLRF and LIP-RTVE drops WER from 59.7% to 24.8% on VLRF, and from roughly 80/95% to 34.5/59.5% on LIP-RTVE SD/SI. Even the lowest-performing full-model variant in the ablation, with no LM, sits far below previous bests. That is a genuine improvement, not a fiddling gain.

What's actually new is not the architecture—it's borrowed from Ma et al. (2022)—but the fine-tuning recipes, the consolidated four-corpus Spanish benchmark (controlled, TV, vlog, TED), the component ablation, and a Zipf-law-based error analysis. The paper is honest about its constraints: single GPU, batch size 1, no auxiliary-task pre-training, and it flags these in the limitations section. That transparency earns credit.

Soft spots, in proportion. First, every configuration is a single training run. The bootstrap CIs only capture test-set sampling, not seed variance. For the big SOTA claims this doesn't matter—you'd need implausibly large run-to-run variance to erase a 20-45 point gap. But the smaller ablation deltas (e.g., data augmentation seems to hurt by 0.9-2.2 points) are within noise and should be read as suggestive. Second, the LM for LIP-RTVE is fine-tuned on contemporary TV-newscast text, which is in-domain; that likely inflates the LM contribution, though the LM-free rows still beat prior SOTA. Third, the Zipf analysis is descriptive. The coverage statistics correlate with WER across corpora, but the authors don't claim causation, and the comparison of VLRF vs LIP-RTVE SD shows other factors dominate. Fine.

Who should read it: anyone working on lipreading for low-resource languages, or on Spanish VSR specifically. It's a useful baseline consolidator and an honest ablation. If this crossed my desk as an editor, I'd send it to review—the empirical margins are important even if the novelty is incremental. I'd ask the authors for either multiple seeds or a clear statement that the reported deltas are single-run point estimates, and to double-check LM fine-tuning leakage. But those are revision-level requests, not grounds for rejection.

My take: if you're in this area, cite it and use the benchmark. Not a game-changer, but a solid, trustworthy step forward.

Referee Report

2 major / 6 minor

Summary. The manuscript presents an end-to-end continuous Spanish lipreading system based on the hybrid CTC/Attention architecture of Ma et al. (2022), pre-trained on more than 1500 hours of English data and fine-tuned on Spanish corpora. It reports word error rates of 24.8% on VLRF (speaker-dependent), 34.5% on LIP-RTVE (speaker-dependent), and 59.5% on LIP-RTVE (speaker-independent), compared with previous bests of approximately 59.7%, 80%, and 95%, respectively. The paper also presents an ablation study covering data augmentation, language-model fine-tuning, language-model removal, and the CTC and attention decoder branches; an error analysis with WER histograms, Zipf-law statistics, and vocabulary-coverage measures; and a proposed four-corpus Spanish benchmark that includes CMU-MOSEAS es and MuA ViC es.

Significance. If the reported results are reproducible, this is a substantial advance for Spanish visual speech recognition, with large absolute WER reductions on two corpora of very different recording conditions. The paper is commendably transparent: it reports bootstrap confidence intervals, states GPU-related limitations, releases code and trained models, and clearly marks results taken from Ma et al. (2022) rather than re-run. The qualitative state-of-the-art claim is robust because the reported margins (24.8 vs. 59.7 on VLRF; 34.5 vs. ~80 and 59.5 vs. ~95 on LIP-RTVE) are far larger than plausible training-stochasticity effects. The main weakness is that the ablation study, which is a stated contribution, is based on a single training run per configuration and several ablation deltas fall within the reported confidence intervals; this affects component-level conclusions but not the headline results.

major comments (2)

[Section 6, Table 2] The ablation conclusions in Table 2 are not statistically supported because each configuration is a single training run and the bootstrap intervals quantify only test-set sampling noise, not training stochasticity. For example, removing data augmentation changes WER from 24.8±3.4 to 23.9±3.4 on VLRF and from 34.5±1.2 to 32.3±1.2 on LIP-RTVE SD, both deltas being smaller than the reported interval half-widths. The statements that data augmentation "slightly hinders" learning and the relative ordering of the LM fine-tuning effects therefore cannot be distinguished from random seed variation. Please provide repeated-seed results or explicitly reframe these component-level conclusions as preliminary observations rather than definite findings.
[Section 5.1 and Section 6] All hyperparameters (α=0.1, λ=0.1, β=0.4, learning rates, epoch counts, beam size, and random crop size) are inherited from Ma et al. (2022) and are not tuned on VLRF or LIP-RTVE. Consequently, the phrase "best setting along our experiments" in Section 6 is too strong: the configuration is the best among the tested ablations, not a tuned optimum. The large SOTA margins are probably insensitive to this, but the ablation deltas and the ranking of configurations could change under a targeted hyperparameter search. The manuscript should state explicitly that no hyperparameter search was performed on the target datasets.

minor comments (6)

[Table 2, footnote] The row "− Fine-Tuning >100†" is used to argue that transfer without fine-tuning fails, but the footnote only says "due to a peculiarity of the WER metric." Please explain concretely, for instance that insertion-heavy hypotheses can push WER above 100%, so that readers can interpret the magnitude.
[Section 4] The text "one the first multi-lingual datasets" is missing "of," and the corpus names are inconsistent (LIP-RTVE versus LIP-R TVE, MuA ViC versus MuA ViCes). Please standardize the names throughout.
[Section 5.1] It is unclear whether the "nearly 300k sentences" used for VLRF LM fine-tuning are the VLRF transcripts themselves or an external text corpus; please clarify the source and its relationship to the VLRF test set.
[Table 5] Since the paper cites Yeo et al. (2024) as surpassing the state of the art on MuA ViC with around 46% WER, the benchmark table should either include that result or explicitly note that only Ma et al. (2022) results are listed for comparability.
[Figure 5] The caption states that both axes are logarithmic, but the axis labels use powers of two without axis titles; adding axis titles and units would improve readability.
[Abstract and Conclusions] The term "rigorous" for the error analysis is somewhat strong given the absence of statistical tests on the Zipf-law and vocabulary-coverage comparisons; "detailed" would be more measured.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: WER claims are empirical measurements on held-out partitions; self-citations are baseline context only.

full rationale

The paper's central claims are experimental. The SOTA margins for VLRF and LIP-RTVE are obtained by running an end-to-end CTC/Attention system on official train/validation/test splits and measuring WER on held-out test sets; the comparison baselines (59.7% WER on VLRF, roughly 80%/95% on LIP-RTVE) come from earlier published work, including the authors' own, but are not inputs to the current model and do not define the new WERs. The transfer dependence on Ma et al. (2022) weights is explicitly probed with the '− Fine-Tuning' rows, which show poor performance without target-domain fine-tuning, so the reported gains are not forced by the pretrained initialization. The ablation study removes one component at a time from a trained system and measures the resulting WER; the conclusion that CTC is important follows from measured degradation (e.g., 51.3 vs 24.8 WER on VLRF), not from definition. Self-citations to Gimeno-Gómez and Martínez-Hinarejos (2022, 2024) supply the prior baselines and corpus description; they are context, not load-bearing justification for the new results. The only substantive caveats—single training runs without seed variation, and LM fine-tuning on in-domain TV-newscast text—affect statistical strength and attribution of the LM contribution, but they are not circularity: the held-out WERs remain independent measurements. No equation in the paper defines a predicted quantity in terms of the fitted quantity being reported, and no uniqueness or ansatz claim is imported from self-citations.

Assumptions & free parameters 11 free parameters · 6 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The central results rest on standard deep-learning components, hyperparameters inherited from prior work, and domain-level assumptions about transfer learning, data splits, and the value of Zipf-based error analysis. The free parameters are all hyperparameters chosen by hand or copied from Ma et al. (2022).

free parameters (11)

CTC/Attention loss weight alpha = 0.1
Equation (1); set to 0.1 per Ma et al. (2022), not tuned on the target datasets.
Inference CTC/Attention weight lambda = 0.1
Equation (2); set per Ma et al. (2022) for beam search.
Language model weight beta = 0.4
Equation (2); set per Ma et al. (2022), and set to 0.0 when the LM is discarded.
Word insertion penalty = 0.0
Section 5.1; chosen by hand, affects decoding.
Beam size = 10
Section 5.1; chosen by hand for beam search.
Learning rate for VLRF fine-tuning = 5e-4
Section 5.1; fixed for the VLRF experiments.
Learning rate for LIP-RTVE SD and SI = 5e-4 and 5e-5
Section 5.1; assigned per speaker-dependent and speaker-independent scenario.
Training epochs = 5
Section 5.1; fixed for all VSR fine-tuning runs.
LM fine-tuning learning rate = 5e-5
Section 5.1; fixed for language model fine-tuning.
Random crop size = 88x88 pixels
Section 3.1; data augmentation choice inherited from prior work.
Vocabulary size = 37 symbols
Section 3.1; character-level tokenizer size including special tokens, following Ma et al. (2022).

assumptions (6)

domain assumption English-pretrained visual representations from Ma et al. (2022) transfer to Spanish lipreading after fine-tuning.
Section 5.1; the entire method depends on this transfer learning assumption, and the no-fine-tuning ablation shows it is essential.
standard math The bootstrap method of Bisani and Ney (2004) gives valid 95% confidence intervals for WER on the reported test sets.
Section 5.1; used for all confidence intervals.
domain assumption Removing one component in the ablation isolates that component's contribution.
Section 5.2; potential interactions between components are not modeled.
domain assumption Zipf's law is a meaningful diagnostic for VSR error analysis on these corpora.
Section 6; used to explain vocabulary coverage and out-of-vocabulary effects.
domain assumption The official data splits for VLRF, LIP-RTVE, CMU-MOSEAS, and MuA ViC are correct and comparable.
Section 4; the benchmark reliability depends on these splits.
domain assumption The LM fine-tuning text collections (300k VLRF sentences, 80k LIP-RTVE sentences) are appropriate domain-adapted corpora.
Section 5.1; the LM gains depend on these collections being representative.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions." pith.science (2026). https://pith.science/paper/JZYNZHZT

@misc{pith2026250200464,
  author       = {Pith},
  title        = {Pith review of: Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JZYNZHZT}},
  note         = {Machine review of arXiv:2502.00464}
}

read the original abstract

Visual speech recognition remains an open research problem where different challenges must be considered by dispensing with the auditory sense, such as visual ambiguities, the inter-personal variability among speakers, and the complex modeling of silence. Nonetheless, recent remarkable results have been achieved in the field thanks to the availability of large-scale databases and the use of powerful attention mechanisms. Besides, multiple languages apart from English are nowadays a focus of interest. This paper presents noticeable advances in automatic continuous lipreading for Spanish. First, an end-to-end system based on the hybrid CTC/Attention architecture is presented. Experiments are conducted on two corpora of disparate nature, reaching state-of-the-art results that significantly improve the best performance obtained to date for both databases. In addition, a thorough ablation study is carried out, where it is studied how the different components that form the architecture influence the quality of speech recognition. Then, a rigorous error analysis is carried out to investigate the different factors that could affect the learning of the automatic system. Finally, a new Spanish lipreading benchmark is consolidated. Code and trained models are available at https://github.com/david-gimeno/evaluating-end2end-spanish-lipreading.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 52 canonical work pages

[1]

Acosta-Triana, J.-M., Gimeno-G´ omez, D., Mart ´ ınez-Hinarejos, C.-D. (2024). AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies. Proc. of LREC-COLING (pp. 1260–1269)

work page 2024
[2]

Afouras, T., Chung, J.-S., Zisserman, A. (2018). LRS3-TED: a large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496 , , Retrieved from https://doi.org/10.48550/arXiv.1809.00496

arXiv doi:10.48550/arxiv.1809.00496 2018
[3]

Anwar, M., Shi, B., Goswami, V., Hsu, W., Pino, J., Wang, C. (2023). MuA ViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation. Interspeech (pp. 4064–4068)

work page 2023
[4]

Weber, G

Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., . . . Weber, G. (2020). Common voice: A massively-multilingual speech corpus. Proc. LREC (pp. 4218–4222). Retrieved from https://aclanthology.org/2020.lrec-1.520

work page 2020
[5]

Baevski, A., Zhou, Y., Mohamed, A., Auli, M. (2020). wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural 17 information processing systems , 33 , 12449–12460, https://doi.org/10.5555/ 3495724.3496768

arXiv 2020
[6]

Bear, H., & Harvey, R. (2016). Decoding visemes: Improving machine lip-reading. ICASSP (pp. 2009–2013). Retrieved from https://doi.org/10.1109/ICASSP .2016.7472029

arXiv 2016
[7]

Bear, H., Harvey, R., Theobald, B., Lan, Y. (2014a). Resolution limits on visual speech recognition. ICIP (pp. 1371–1375). Retrieved from https://doi.org/10.1109/ ICIP.2014.7025274

work page arXiv 2014
[8]

Besle, J., Fort, A., Delpuech, C., Giard, M.-H. (2004). Bimodal speech: early suppres- sive visual effects in human auditory cortex. European journal of Neuroscience, 20 (8), 2225–2234, https://doi.org/10.1111%2Fj.1460-9568.2004.03670.x

work page arXiv 2004

Show all 72 references

[9]

Bisani, M., & Ney, H. (2004). Bootstrap estimates for confidence intervals in asr performance evaluation. Icassp (Vol. 1, pp. 409–412). Retrieved from https:// doi.org/10.1109/ICASSP.2004.1326009

2004 arXiv
[10]

Bowden, R., Cox, S., Harvey, R., Lan, Y., Ong, E.-J., Owen, G., Theobald, B.-J. (2013). Recent developments in automated lip-reading. Optics and Photonics for

2013
[11]

Bulat, A., & Tzimiropoulos, G. (2017). How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). Iccv (p. 1021-1030). Retrieved from https://doi.org/10.1109/ICCV.2017.116

2017 doi
[12]

Campbell, R. (2008). The Processing of Audio-Visual Speech: Empirical and Neural Bases. Philosophical Transactions of the Royal Society B: Biological Sciences , 363 (1493), 1001–1010,

2008
[13]

Chang, O., Liao, H., Serdyuk, D., Shah, A., Siohan, O. (2024). Conformer is All You Need for Visual Speech Recognition. Icassp (p. 10136-10140)

2024
[14]

Chung, J., & Zisserman, A. (2017). Lip reading in the wild. 13th asian conference on computer vision (pp. 87–103). 18

2017
[15]

Cox, S.J., Harvey, R.W., Lan, Y., Newman, J.L., Theobald, B.-J. (2008). The challenge of multispeaker lip-reading. AVSP (pp. 179–184). Retrieved from https:// www.isca-speech.org/archive open/avsp08/av08 179.html

2008
[16]

Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., Salakhutdinov, R. (2019). Transformer-XL: Attentive language models beyond a fixed-length context.Proc. of the 57th acl (pp. 2978–2988). ACL

2019
[17]

Deng, J., Guo, J., Ververas, E., Kotsia, I., Zafeiriou, S. (2020). Retinaface: Single-shot multi-level face localisation in the wild. CVPR (p. 5202-5211). Retrieved from 10.1109/CVPR42600.2020.00525

2020
[18]

Dungan, L., Karaali, A., Harte, N. (2018). The impact of reduced video quality on visual speech recognition. ICIP (p. 2560-2564). Retrieved from http://doi.org/ 10.1109/ICIP.2018.8451754

2018
[19]

Egorov, E., Kostyumov, V., Konyk, M., Kolesnikov, S. (2021). LR WR: large- scale benchmark for lip reading in Russian language. arXiv preprint arXiv:2109.06692 , ,

2021 arXiv
[20]

Ezz, M., Mostafa, A.M., Nasr, A.A. (2020). A Silent Password Recognition Framework Based on Lip Analysis. IEEE Access, 8 , 55354-55371,

2020
[21]

Feng, Z. (2023). Formal analysis for natural language processing: A handbook. Springer Nature. Retrieved from https://doi.org/10.1007/978-981-16-5172-4

2023 doi
[22]

Fernandez-Lopez, A., Chen, H., Ma, P., Haliassos, A., Petridis, S., Pantic, M. (2023). Sparsevsr: Lightweight and noise robust visual speech recognition.arXiv preprint arXiv:2307.04552 , ,

2023 arXiv
[23]

Fernandez-Lopez, A., Martinez, O., Sukno, F.M. (2017). Towards estimating the upper bound of visual-speech recognition: The visual lip-reading feasibility database. 12th fg (pp. 208–215). Retrieved from https://doi.org/10.1109/FG.2017.34

2017 doi
[24]

Fernandez-Lopez, A., & Sukno, F.M. (2018). Survey on automatic lip-reading in the era of deep learning. Image and Vision Computing , 78 , 53–72, https://doi.org/ https://doi.org/10.1016/j.imavis.2018.07.002

2018 doi
[25]

Fernandez-Lopez, A., & Sukno, F.M. (2022). End-to-End Lip-Reading Without Large- Scale Data. IEEE/ACM TASLP , 30 , 2076-2090, https://doi.org/10.1109/ TASLP.2022.3182274 19 Fern´ andez-L´ opez, A., & Sukno, F. (2017). Optimizing phoneme-to-viseme mapping for continuous lip-rea...

2022
[26]

Gales, M., & Young, S. (2008). The application of hidden markov models in speech recognition. Now Publishers Inc. Retrieved from http://doi.org/10.1561/ 2000000004 Gimeno-G´ omez, D., & Mart ´ ınez-Hinarejos, C.-D. (2024). Continuous lipreading based on acoustic temporal align...

2008
[27]

Graves, A., Fern´ andez, S., Gomez, F., Schmidhuber, J. (2006). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural net- works. 23rd icml (p. 369–376). ACM. Retrieved from https://doi.org/10.1145/ 1143844.1143891

2006
[28]

Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., . . . Pang, R. (2020). Conformer: Convolution-augmented Transformer for Speech Recog- nition. Proc. Interspeech (pp. 5036–5040). Retrieved from https://doi.org/ 10.21437/Interspeech.2020-3015

2020 doi
[29]

Haliassos, A., Zinonos, A., Mira, R., Petridis, S., Pantic, M. (2024). BRA VEn: Improv- ing Self-supervised pre-training for Visual and Auditory Speech Recognition. Icassp (p. 11431-11435)

2024
[30]

Harte, N., & Gillen, E. (2015). TCD-TIMIT: An audio-visual corpus of continuous speech. IEEE Transactions on Multimedia , 17 (5), 603–615, https://doi.org/ 10.1109/TMM.2015.2407694

2015
[31]

He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. CVPR (p. 770-778). Retrieved from https://doi.org/10.1109/CVPR.2016.90

2016 doi
[32]

Higuchi, Y., Inaguma, H., Watanabe, S., Ogawa, T., Kobayashi, T. (2021). Improved Mask-CTC for Non-Autoregressive End-to-End ASR. ICASSP (p. 8363-8367)

2021
[33]

Ivanko, D., Ryumin, D., Karpov, A. (2019). Automatic Lip-Reading of Hearing Impaired People. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences , XLII-2/W12 , 97–101, https://doi .org/10.5194/isprs-archives-XLII-2-W12-97-2019 20

2019 doi
[34]

Jha, A., Namboodiri, V.P., Jawahar, C.V. (2019). Spotting Words in Silent Speech Videos: A Retrieval-Based Approach. Machine Vision and Applications , 30 , 217–229,

2019
[35]

Kim, M., Yeo, J.H., Choi, J., Ro, Y.M. (2023). Lip reading for low-resource languages by learning and combining general speech knowledge and language-specific knowledge. Iccv (pp. 15359–15371)

2023
[36]

Koller, O., Forster, J., Ney, H. (2015). Continuous Sign Language Recognition: Towards Large Vocabulary Statistical Recognition Systems Handling Multiple Signers. Computer Vision and Image Understanding , 141 , 108–125, https:// doi.org/10.1016/j.cviu.2015.09.013

2015 doi
[37]

Laux, H., Hallawa, A., Assis, J.C.S., Schmeink, A., Martin, L., Peine, A. (2023). Two- Stage Visual Speech Recognition for Intensive Care Patients. Scientific Reports, 13 (1), 928,

2023
[38]

Lee, J., & Watanabe, S. (2021). Intermediate Loss Regularization for CTC-Based Speech Recognition. ICASSP (p. 6224-6228)

2021
[39]

Liao, J., Duan, H., Feng, K., Zhao, W., Yang, Y., Chen, L. (2023). A Light Weight Model for Active Speaker Detection. Proc. of the ieee/cvf cvpr (p. 22932-22941)

2023
[40]

Fuegen, C

Liu, X., Lakomkin, E., Vougioukas, K., Ma, P., Chen, H., Xie, R., . . . Fuegen, C. (2023). SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision. Cvpr (pp. 18806–18815)

2023
[41]

Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. Iclr. Retrieved from https://openreview.net/pdf?id=Bkg6RiCqY7

2019
[42]

Ma, P., Haliassos, A., Fernandez-Lopez, A., Chen, H., Petridis, S., Pantic, M. (2023). Auto-avsr: Audio-visual speech recognition with automatic labels. Icassp (p. 1- 5)

2023
[43]

Ma, P., Petridis, S., Pantic, M. (2021). End-to-end audio-visual speech recognition with conformers. ICASSP (p. 7613-7617). Retrieved from https://doi.org/10 .1109/ICASSP39728.2021.9414567

2021
[44]

Ma, P., Petridis, S., Pantic, M. (2022). Visual speech recognition for multiple languages in the wild. Nature Machine Intelligence , 4 (11), 930–939, https://doi.org/ 10.1038/s42256-022-00550-z 21

2022 doi
[45]

Manaris, B., Pellicoro, L., Pothering, G., Hodges, H. (2006). Investigating esperanto’s statistical proportions relative to other languages using neural networks and zipf’s law. Proceedings of the 24th iasted international conference on artificial intelligence and applications...

2006
[46]

McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264 (5588), 746–748, https://doi.org/10.1038/264746a0

1976 doi
[47]

McAuley, D.F. (2023). A user evaluation of speech/phrase recognition software in critically ill patients: A decide-ai feasibility study. Critical Care, 27 (1), 277,

2023
[48]

Ott, M., Edunov, S., Grangier, D., Auli, M. (2018). Scaling neural machine translation. Proc. of the 3rd conference on machine translation (pp. 1–9). ACL. Retrieved from https://doi.org/10.18653/v1/W18-6301

2018 doi
[49]

Park, S.J., Kim, C.W., Rha, H., Kim, M., Hong, J., Yeo, J., Ro, Y.M. (2024). Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation. Proc. of the 62nd acl (pp. 16334–16348)

2024
[50]

Piantadosi, S.T. (2014). Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review , 21 , 1112–1130, https://doi.org/10.3758/s13423-014-0585-6

2014 doi
[51]

Potamianos, G., Neti, C., Gravier, G., Garg, A., Senior, A. (2003). Recent advances in the automatic recognition of audiovisual speech. Proc. of the IEEE , 91 (9), 1306–1326, https://doi.org/10.1109/JPROC.2003.817150

2003
[52]

Prajwal, K., Mukhopadhyay, R., Namboodiri, V., Jawahar, C.V. (2020). A lip sync expert is all you need for speech to lip generation in the wild. Proceedings of the 28th acm international conference on multimedia (pp. 484–492)

2020
[53]

Prajwal, K.R., Afouras, T., Zisserman, A. (2022). Sub-word level lip read- ing with visual attention. CVPR (p. 5162-5172). Retrieved from https://openaccess.thecvf.com/content/CVPR2022/html/Prajwal Sub-Word Level Lip Reading With Visual Attention CVPR 2022 paper.html

2022
[54]

Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., Collobert, R. (2020). MLS: A Large- Scale Multilingual Dataset for Speech Research. Proc. interspeech (pp. 2757– 2761). Retrieved from https://doi.org/10.21437/Interspeech.2020-2826 22

2020 doi
[55]

Ramachandran, P., Zoph, B., Le, Q.V. (2017). Searching for activation functions. arXiv preprint arXiv:1710.05941 , , Retrieved from https://arxiv.org/abs/1710 .05941

2017 arXiv
[56]

Post, M. (2021). The Multilingual TEDx Corpus for Speech Recognition and Translation. Proc. interspeech (pp. 3655–3659). Retrieved from https//doi.org/ 10.21437/Interspeech.2021-11

2021 doi
[57]

Shi, B., Hsu, W.N., Lakhotia, K., Mohamed, A. (2022). Learning audio- visual speech representation by masked multimodal cluster prediction. arXiv preprint arXiv:2201.02184 , , Retrieved from https://doi.org/10.48550/arXiv .2201.02184

2022 doi arXiv
[58]

Smith, L.N., & Topin, N. (2019). Super-convergence: very fast training of neural networks using large learning rates. Ai and ml for multi-domain operations applications (Vol. 11006, pp. 369–386). Retrieved from https://doi.org/10.1117/ 12.2520589 Son Chung, J., Senior, A., Vin...

2019
[59]

Stafylakis, T., & Tzimiropoulos, G. (2018). Zero-Shot Keyword Spotting for Visual Speech Recognition In-the-Wild. Proc. of eccv (pp. 513–529)

2018
[60]

Tao, R., Pan, Z., Das, R., Qian, X., Shou, M., Li, h. (2021). Is Someone Speak- ing? Exploring Long-Term Temporal Features for Audio-Visual Active Speaker Detection. Proc. of the 29th acm international conference on multimedia (p. 3927–3935). Association for Computing Machinery

2021
[61]

Thangthai, K. (2018). Computer lipreading via hybrid deep neural network hidden markov models (Doctoral dissertation, University of East Anglia). Retrieved from https://ueaeprints.uea.ac.uk/id/eprint/69215

2018
[62]

Theobald, B.J., Harvey, R., Cox, S.J., Lewis, C., Owen, G.P. (2006). Lip-Reading Enhancement for Law Enforcement. Optics and photonics for counterterrorism and crime fighting ii (Vol. 6402, pp. 24–32)

2006
[63]

Polo- sukhin, I

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., . . . Polo- sukhin, I. (2017). Attention is all you need. NeurIPS , 30 , 6000–6010, Retrieved from https://dl.acm.org/doi/10.5555/3295222.3295349 23

2017
[64]

Ochiai, T

Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., . . . Ochiai, T. (2018). ESPnet: End-to-End Speech Processing Toolkit. Proc. interspeech (pp. 2207–2211). Retrieved from https://doi.org/10.21437/Interspeech.2018-1456

2018 doi
[65]

Watanabe, S., Hori, T., Kim, S., Hershey, J.R., Hayashi, T. (2017). Hybrid ctc/at- tention architecture for end-to-end speech recognition. IEEE JSTSP , 11 (8), 1240-1253, https://doi.org/10.1109/JSTSP.2017.2763455

2017
[66]

Wei, G., Duan, Z., Li, S., Yang, G., Yu, X., Li, J. (2023). Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech Recognition. arXiv preprint arXiv:2304.04991 , ,

2023 arXiv
[67]

Yang, S., Zhang, Y., Feng, D., Yang, M., Wang, C., Xiao, J., . . . Chen, X. (2019). LR W-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild. 14th ieee international conference on automatic face & gesture recognition (p. 1-8)

2019
[68]

Yeo, J.H., Kim, M., Watanabe, S., Ro, Y.M. (2024). Visual speech recognition for languages with limited labeled data using automatic labels from whisper. Icassp (p. 10471-10475)

2024
[69]

Zadeh, A.B., Cao, Y., Hessner, S., Liang, P.P., Poria, S., Morency, L.-P. (2020). CMU- MOSEAS: A multimodal language dataset for spanish, portuguese, german and french. EMNLP (pp. 1801–1812). Retrieved from https://doi.org/10.18653/v1/ 2020.emnlp-main.141

2020 doi
[70]

Zhang, Y., Yang, S., Xiao, J., Shan, S., Chen, X. (2020). Can we read speech beyond the lips? rethinking roi selection for deep visual speech recognition. 15th ieee fg (p. 356-363)

2020
[71]

Zipf, G.K. (1936). The psychobiology of language. Houghton, Mifflin. Retrieved from https://psycnet.apa.org/record/1935-04756-000

1936
[72]

Zipf, G.K. (1949). Human behavior and the principle of least effort. Addison-Wesley Press. Retrieved from https://psycnet.apa.org/record/1950-00412-000 24

1949

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Acosta-Triana, J.-M., Gimeno-G´ omez, D., Mart ´ ınez-Hinarejos, C.-D. (2024). AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies. Proc. of LREC-COLING (pp. 1260–1269)

work page 2024

[2] [2]

Afouras, T., Chung, J.-S., Zisserman, A. (2018). LRS3-TED: a large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496 , , Retrieved from https://doi.org/10.48550/arXiv.1809.00496

arXiv doi:10.48550/arxiv.1809.00496 2018

[3] [3]

Anwar, M., Shi, B., Goswami, V., Hsu, W., Pino, J., Wang, C. (2023). MuA ViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation. Interspeech (pp. 4064–4068)

work page 2023

[4] [4]

Weber, G

Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., . . . Weber, G. (2020). Common voice: A massively-multilingual speech corpus. Proc. LREC (pp. 4218–4222). Retrieved from https://aclanthology.org/2020.lrec-1.520

work page 2020

[5] [5]

Baevski, A., Zhou, Y., Mohamed, A., Auli, M. (2020). wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural 17 information processing systems , 33 , 12449–12460, https://doi.org/10.5555/ 3495724.3496768

arXiv 2020

[6] [6]

Bear, H., & Harvey, R. (2016). Decoding visemes: Improving machine lip-reading. ICASSP (pp. 2009–2013). Retrieved from https://doi.org/10.1109/ICASSP .2016.7472029

arXiv 2016

[7] [7]

Bear, H., Harvey, R., Theobald, B., Lan, Y. (2014a). Resolution limits on visual speech recognition. ICIP (pp. 1371–1375). Retrieved from https://doi.org/10.1109/ ICIP.2014.7025274

work page arXiv 2014

[8] [8]

Besle, J., Fort, A., Delpuech, C., Giard, M.-H. (2004). Bimodal speech: early suppres- sive visual effects in human auditory cortex. European journal of Neuroscience, 20 (8), 2225–2234, https://doi.org/10.1111%2Fj.1460-9568.2004.03670.x

work page arXiv 2004