Pith. sign in

REVIEW 4 major objections 4 minor 40 references

RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modeling

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RESOUND reconstructs intelligible, natural speech from silent talking-face video by decomposing generation into acoustic and semantic branches rooted in source–filter theory.

desk verdict A genuinely new L2S architecture with a real evaluation leak: the ground-truth audio prompt and the Auto-AVSR circularity undermine the SOTA claim until re-baselined. read the letter →

arxiv 2505.22024 v1 pith:VDKR2PHU submitted 2025-05-28 cs.SD cs.CVeess.AS

Long-Khanh Pham , Thanh V. T. Tran , Minh-Tan Pham , Van Nguyen This is my paper · ORCID
classification cs.SDcs.CVeess.AS
keywords lip-to-speechsynthesissource-filtertheoryacoustic-semanticdecompositionspeechunitsprosodypredictionmultimodalfusionaudio-visualrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Lip-to-speech synthesis must recover both what was said and how it was said from a silent face. RESOUND claims that these two jobs should be split, following source–filter theory: an acoustic branch predicts prosody — pitch, energy, timbre — from a short speaker prompt, while a semantic branch derives linguistic content from lip movements, lip-to-text predictions, and phonemes. The paper reports that on LRS3-TED this split lowers word error rate to 20.06 from 27.69 for the best compared baseline, raises naturalness to 3.002 UTMOS, and runs at near-real-time speed. If right, the work says that deliberately separating source and filter representations makes lip-to-speech easier to optimize and more practical.

What carries the argument

The load-bearing mechanism is a dual-path architecture inspired by source–filter theory. The Excitation Generator models the glottal source (pitch, energy, timbre, and a speaker prompt) while the Formant Generator models the vocal-tract filter (visual features and phonemes); their outputs are summed into a coarse mel-spectrogram that a Mel-Decoder refines, and a Linguistic Predictor emits discrete speech units that condition a vocoder alongside the mel-spectrogram. The semantic side uses AV-HuBERT to extract visual-linguistic features, a lip-to-text model whose output is converted by G2P into phoneme embeddings, and a Semantic Reference Attention module that aligns visual and phonetic sequences with a Reference Transformer followed by a Conformer. The prosody side uses a timbre encoder, a Speaker Prompt Encoder, and FastSpeech-2-style pitch and energy predictors, so each branch can be optimized independently.

What would settle it

Run the paper's MOS protocol with human listeners on RESOUND's LRS3-TED outputs and decode the same outputs with a second, independently trained automatic speech recognizer; if the human ratings or the second recognizer's WER do not reproduce the advantage over LipVoicer and DiffV2S shown in Table 1, the intelligibility claim is partly an artifact of using the same recognizer for supervision and evaluation.

Watch

Extended reading notes

Core claim

RESOUND's central claim is that decomposing speech generation into an excitation (acoustic/prosodic) representation and a formant (semantic/linguistic) representation, then fusing them additively into a coarse mel-spectrogram, yields state-of-the-art lip-to-speech quality. On LRS3-TED the system reports 20.06 WER, 3.002 UTMOS, 0.777 SECS, and 0.423 ESTOI, beating the compared systems on most metrics; on LRS2-BBC it improves most metrics except WER (28.55 versus 17.04 for LipVoicer), which the authors attribute to noise in that dataset. Ablations show that removing the lip-to-text branch and the Semantic Reference Attention raises WER by roughly 37%, and removing the acoustic branch degrades naturalness by about 28% and pitch error by about 40%. The acoustic branch also predicts pitch and energy explicitly, which the paper argues improves prosody beyond prior source-filter work that ignored energy.

Load-bearing premise

The load-bearing assumption is that the Auto-AVSR word-error rate is an unbiased measure of intelligibility even though the same Auto-AVSR model supplies the text predictions used as training supervision in the semantic branch.

Editorial extensions

If this is right

  • On LRS3-TED, RESOUND reaches 20.06 WER against 27.69 for the best compared baseline, with 3.002 UTMOS and 0.777 SECS; if the measurements hold, it is the best reported lip-to-speech result on that benchmark.
  • With RTF 0.063, it generates one second of speech in about 63 milliseconds and runs more than 200 times faster than LipVoicer (RTF 14.117), making real-time use plausible.
  • Ablations show that removing the lip-to-text branch and the Semantic Reference Attention raises WER by about 37%, while removing the acoustic branch lowers naturalness by about 28%; the two branches contribute independently.
  • On LRS2-BBC, the system improves most metrics but not WER (28.55 versus 17.04 for LipVoicer); the paper attributes the gap to noise in LRS2 and to LipVoicer trading naturalness for content accuracy.
  • When the vocoder is fed ground-truth mel-spectrograms and speech units, performance nearly closes the gap to ground truth (UTMOS 3.43, WER 1.21), indicating the encoding side, not the vocoder, carries most of the reconstruction burden.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same recognizer (Auto-AVSR) both supplies the lip-to-text supervision and computes the reported WER, the cleanest test of the intelligibility claim is a human listening study or evaluation with a second, independently trained recognizer; the paper's numbers alone leave this overlap unsettled.
  • The explicit acoustic/semantic split suggests the model could edit prosody and content independently (for example, changing emotion while keeping the words), a capability the paper does not demonstrate.
  • Using a 0.5-second audio prompt for timbre together with visual content implies a possible path to voice cloning from silent video, with privacy implications the paper does not discuss.
  • The weaker LRS2 WER indicates the method's robustness in noisy, uncontrolled conditions is not yet established, so the general claim about real-world applicability should be read as strongest for relatively clean TED-style speech.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes RESOUND, a lip-to-speech (L2S) system that decomposes speech reconstruction into an acoustic branch (prosody, timbre) and a semantic branch (linguistic content) motivated by source-filter theory. The acoustic branch predicts pitch and energy from a short audio prompt, while the semantic branch combines visual features from AV-HuBERT with text predictions from an L2T model and phoneme embeddings. The two streams are fused in a Spec-Ling Decoder that predicts both a mel-spectrogram and discrete speech units, followed by a vocoder. Experiments on LRS2-BBC and LRS3-TED report improved WER, UTMOS, SECS, and low-level metrics over prior methods, with ablations supporting the individual contributions. The manuscript includes a demo page and states that source code will be released.

Significance. If the results hold, RESOUND would be a valuable contribution: it is the first L2S system to explicitly apply source-filter decomposition, it combines multiple supervision signals (visual, phoneme, speech units, acoustic predictors) in a structured way, and it improves objective and subjective quality on large-scale datasets. The code release and the ablation studies are assets. However, the evaluation contains two load-bearing confounds: the WER metric uses the same ASR model that provides the L2T supervision, and the test-time speaker prompt is a slice of the ground-truth audio. These issues affect the validity of the central SOTA claim and must be addressed before the results can be fully trusted.

major comments (4)
  1. [§3.2, §2.2.1] The intelligibility claim rests on WER computed by Auto-AVSR, which is the same model ([22]) used inside RESOUND as the L2T text predictor. Section 2.2.1 states that the L2T model generates textual predictions that are converted to phoneme embeddings, and these are used to supervise the semantic branch. Section 3.2 then evaluates WER with Auto-AVSR. This creates a circular evaluation: the model is trained to produce speech that this specific recognizer decodes well, and then measured with that same recognizer. The reported improvement from 27.69 to 20.06 on LRS3-TED may therefore reflect an overfit to Auto-AVSR rather than a general gain in intelligibility. The authors should report WER with an independent ASR not used in training (e.g., Whisper or an ESPnet model) and, ideally, human transcription evaluations on a larger set than the 20 samples in Table 2.
  2. [§3.3, Table 1] The test-time setup leaks information from the ground-truth audio. Section 3.3 says a 0.5s segment is randomly extracted from the ground-truth audio to serve as the audio prompt. At inference, the model therefore receives a slice of the exact utterance it must reconstruct. This leaks speaker timbre, pitch/energy, and possibly partial lexical content if the slice overlaps the scored portion. All objective metrics in Table 1 (WER, UTMOS, SECS, MAE_F0, ESTOI, MCD-DTW-SL) can be inflated relative to baselines that do not receive this prompt. The paper states only that its own setup follows [11]; it does not indicate whether the published baseline numbers were obtained under the same prompting condition. The SOTA conclusion is confounded unless all methods are re-run with identical prompts (or no prompt). The authors should either remove the ground-truth prompt, use a prompt from a different utterance/spaker, or re-run all baselines under the exact same condition and report the results with error bars.
  3. [Table 1] Table 1 reports a single set of metric values without standard deviations, confidence intervals, or significance tests. Some differences are small (e.g., WER 20.06 vs 21.04 in LRS3-TED; SECS 0.777 vs 0.750 for RESOUND vs Intelligible L2S) and could easily arise from run-to-run variance. Without multiple seeds or bootstrap intervals, the claim that RESOUND 'consistently outperforms' baselines is not statistically supported. The paper should provide variance estimates and, where appropriate, pairwise significance tests.
  4. [§4.1, Table 1 (LRS2-BBC)] On LRS2-BBC, RESOUND has a substantially worse WER than LipVoicer (28.55 vs 17.04). The text acknowledges this and attributes it to noise in LRS2. But intelligibility is a core objective of L2S reconstruction; a method that loses on the primary intelligibility metric on one of the two benchmarks needs a more careful discussion of the trade-off. The paper should either demonstrate that the WER gap is offset by other metrics in a clear way, or temper the 'state-of-the-art' claim to specify the metric and dataset. As written, the claim in the abstract that the method 'consistently outperforms' is too strong.
minor comments (4)
  1. [§4.1, LSE-* footnote] The footnote in Section 4.1 says LSE-* metrics are omitted because some methods outperform ground truth. This is not a convincing reason on its own; LSE-D, LSE-C, and LSE-S are standard in L2S evaluation and are widely reported even when imperfect. Please either provide the scores and explain the discrepancy, or present an alternative synchronization metric, so readers can assess audio-visual alignment.
  2. [Table 2] Table 2 reports MOS on 20 randomly synthesized samples per method. Please specify how the samples were selected (e.g., random utterance indices, seed), how many participants per condition, and whether the 95% confidence intervals are per-method. With only 20 samples and 30 participants, the results are informative but not very precise for a claimed SOTA difference.
  3. [§2.4] Equation (3) lists the loss weights, but there is no sensitivity analysis for these weights (λ_m=100, λ_p=0.1, λ_e=0.1, λ_u=0.01). Given that the paper argues for the importance of the acoustic branch and speech units, a brief experiment on the loss scales would strengthen the robustness of the method.
  4. [§3.3] The implementation details mention random cropping and flipping for the visual input, which are standard augmentations, but there is no mention of an inference-time augmentation or test-time averaging. Please specify the exact inference protocol (e.g., whether the random cropping is disabled at test time).

Circularity Check

2 steps flagged · score 6.0 of 10

WER is measured by the same Auto-AVSR model that supplies RESOUND's text supervision, and the test-time 'speaker prompt' is a 0.5s slice of the ground-truth audio, making the headline gains partly self-referential and partly leak the target.

  1. fitted input called prediction [Section 2.2.1 (Content Extractor) and Section 3.2 (Evaluation Metrics)]
    "we integrate an advanced L2T model [22] to generate textual predictions ... WER is derived from Auto-AVSR [22]."

    Auto-AVSR [22] is both the L2T text predictor inside RESOUND's semantic branch and the recognizer used to compute the reported WER. The model is trained to reproduce text that Auto-AVSR produces from the silent video, and then evaluated by how well Auto-AVSR decodes the synthesized audio. The headline WER gain (20.06 vs 27.69) therefore measures agreement with the system's own text source rather than an independent intelligibility signal. Human MOS and UTMOS are independent and partially mitigate this, but the primary content-accuracy claim is not an external measurement.

  2. other [Section 3.3 (Implementation Details)]
    "Following [11], we randomly extract a 0.5s segment from the ground-truth audio to serve as an audio prompt."

    At test time the model receives a slice of the exact audio it is supposed to reconstruct. Section 2.1 states that the reference mel-spectrogram R_mel is extracted from the input audio prompt and is used by the Timbre Encoder and Prosody Encoder. Thus target pitch, energy, timbre, and possibly partial lexical content are fed into the generator. The objective metrics in Table 1 (WER, UTMOS, SECS, MAE_F0, ESTOI) are then scored on speech conditioned on the ground-truth target, so the SOTA comparison does not measure reconstruction from silent video alone. The paper does not report giving the identical prompt to all baselines, so the comparison is confounded.

full rationale

RESOUND's central intelligibility claim is evaluated with Auto-AVSR [22], the same model that supplies the L2T text predictions used to supervise the semantic branch. This creates an evaluation loop: the synthesized audio is trained to be decodable by Auto-AVSR and is then scored by Auto-AVSR's decoder. The reported WER advantage is therefore not an independent measure of intelligibility, though human MOS and UTMOS provide some external grounding. Additionally, the 0.5s 'speaker prompt' is randomly extracted from the ground-truth audio, so the acoustic branch is conditioned on a slice of the exact target; the Table 1 metrics can be inflated relative to methods that do not receive target audio. No other load-bearing self-citation chain was found. Because the core SOTA claim depends on these two self-referential evaluation choices, the score is 6 rather than 0-2.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on these assumptions: the source-filter mapping is a valid architectural analogy, a ground-truth audio prompt is available, pretrained visual/linguistic models are unbiased, and the ASR-based evaluation faithfully measures intelligibility.

free parameters (5)
  • loss weights lambda_m, lambda_p, lambda_e, lambda_u = 100, 0.1, 0.1, 0.01
    Set by hand in Section 2.4; central to training but not tied to theory.
  • label smoothing alpha = 0.1
    Used in speech unit cross-entropy loss (Section 2.4).
  • speech unit K-means clusters = 200
    HuBERT-BASE with 200 clusters for speech units (Section 3.3).
  • audio prompt length = 0.5s
    Random 0.5s segment from ground-truth audio as speaker prompt (Section 3.3). This is a design choice that requires ground-truth audio at inference.
  • architecture hyperparameters = 8 Conformer layers, 3 FFT blocks, hidden dim 512
    Chosen by hand for the model capacity (Section 3.3).
assumptions (4)
  • domain assumption The source-filter theory decomposition applies to the learned latent representations.
    The paper maps the acoustic branch to glottal source and the semantic branch to vocal tract filter, but no physical source/filter signals are extracted (Section 2, Figure 1).
  • domain assumption A 0.5s ground-truth audio prompt is available at inference.
    Needed for timbre and prosody extraction; a truly silent video cannot provide this audio (Section 3.3).
  • domain assumption AV-HuBERT and L2T models pretrained on LRS3 provide unbiased linguistic features.
    Used as frozen/adapted feature extractors; their training data overlaps with the evaluation datasets (Section 3.3, Section 2.2.1).
  • domain assumption Auto-AVSR WER is a valid measure of intelligibility for the synthesized speech.
    The same model family provides text supervision during training and is used for evaluation (Section 2.2.1 vs Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modeling." pith.science (2026). https://pith.science/paper/VDKR2PHU

@misc{pith2026250522024,
  author       = {Pith},
  title        = {Pith review of: RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VDKR2PHU}},
  note         = {Machine review of arXiv:2505.22024}
}
read the original abstract

Lip-to-speech (L2S) synthesis, which reconstructs speech from visual cues, faces challenges in accuracy and naturalness due to limited supervision in capturing linguistic content, accents, and prosody. In this paper, we propose RESOUND, a novel L2S system that generates intelligible and expressive speech from silent talking face videos. Leveraging source-filter theory, our method involves two components: an acoustic path to predict prosody and a semantic path to extract linguistic features. This separation simplifies learning, allowing independent optimization of each representation. Additionally, we enhance performance by integrating speech units, a proven unsupervised speech representation technique, into waveform generation alongside mel-spectrograms. This allows RESOUND to synthesize prosodic speech while preserving content and speaker identity. Experiments conducted on two standard L2S benchmarks confirm the effectiveness of the proposed method across various metrics.

Figures

Figures reproduced from arXiv: 2505.22024 by the authors.

Figure 1
Figure 1. RESOUND framework: RESOUND decomposes speech generation into acoustic and semantic branches. The acoustic stream extracts prosodic features (pitch, energy, timbre) via the Prosody and Timbre Encoders, while the semantic stream captures content from silent video using the Context Extractor, Mapping Network, and Semantic Reference Attention. The Spec-Ling Decoder then fuses both streams to generate a mel-spectrogram a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 29 canonical work pages

  1. [11]

    Tcd-timit: An audio-visual corpus of continuous speech,

    N. Harte and E. Gillen, “Tcd-timit: An audio-visual corpus of continuous speech,”IEEE Transactions on Multimedia, vol. 17, no. 5, pp. 603–615, 2015

    work page 2015

  2. [22]

    Fant,Acoustic theory of speech production,

    G. Fant,Acoustic theory of speech production,

    work page

  3. [1]

    L2S holds great potential across various domains, from enhancing communication in noisy environments to providing assistive technologies for individuals with aphonia

    Introduction The increasing demand for seamless human-computer interac- tion has driven significant interest in lip-to-speech (L2S) tech- nology due to its broad range of real-world applications. L2S holds great potential across various domains, from enhancing communication in noisy environments to providing assistive technologies for individuals with aph...

    work page

  4. [2]

    Methodology Speech generation from a silent video in RESOUND is de- composed into two branches: an acoustic branch responsible 1https://resound-l2s.github.io/ arXiv:2505.22024v1 [cs.SD] 28 May 2025 Speaker Prompt Excitation Generator Mel-Decoder predicted mel-spectrogram Silent Video Visual Encoder L2T Model text prediction G2P Semantic Reference Attentio...

    work page Pith review arXiv 2025

  5. [3]

    Datasets LRS2-BBC[25] is an English audio-visual dataset from BBC programs, comprising over 220 hours of video

    Experimental Setup 3.1. Datasets LRS2-BBC[25] is an English audio-visual dataset from BBC programs, comprising over 220 hours of video. We use its pre- training and training sets for model development and the test set for inference. LRS3-TED[26] is an English audio-visual dataset from TED/TEDx talks, featuring diverse speakers and a vocabulary of 50,000+ ...

    work page

  6. [4]

    Experimental Results 4.1. Quality Comparison Table 1 presents a comprehensive evaluation of RESOUND on the LRS2 and LRS3 datasets, where our method consistently 2https://github.com/resemble-ai/Resemblyzer 3Unlike prior work, we omit LSE-* metrics as some methods out- perform the ground truth, compromising synchronization reliability. 4https://github.com/Y...

    work page

  7. [5]

    Grounded in source-filter theory, RESOUND separates speech generation into acoustic and semantic branches, capturing prosody and linguistic con- tent

    Conclusion This paper presents RESOUND, a novel framework for recon- structing speech from silent videos. Grounded in source-filter theory, RESOUND separates speech generation into acoustic and semantic branches, capturing prosody and linguistic con- tent. It integrates multiple modalities, utilizing pitch and energy for acoustics and text predictions for...

    work page

  8. [6]

    An audio- visual corpus for speech perception and automatic speech recog- nition,

    M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio- visual corpus for speech perception and automatic speech recog- nition,”The Journal of the Acoustical Society of America, vol. 120, no. 5, pp. 2421–2424, 2006

    2006

Show all 40 references
  1. [7]

    Flow- based unconstrained lip to speech generation,

    J. He, Z. Zhao, Y . Ren, J. Liu, B. Huai, and N. Yuan, “Flow- based unconstrained lip to speech generation,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, pp. 843–851, Jun. 2022. [Online]. Available: https: //ojs.aaai.org/index.php/AAAI/article/...

    2022

  2. [8]

    Let there be sound: Recon- structing high quality speech from silent videos,

    J.-H. Kim, J. Kim, and J. S. Chung, “Let there be sound: Recon- structing high quality speech from silent videos,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 3, 2024, pp. 2759–2767

    2024

  3. [9]

    Lip to speech synthesis with visual context attentional gan,

    M. Kim, J. Hong, and Y . M. Ro, “Lip to speech synthesis with visual context attentional gan,”Advances in Neural Information Processing Systems, vol. 34, pp. 2758–2770, 2021

    2021

  4. [10]

    End-to-end video-to-speech synthesis using gener- ative adversarial networks,

    R. Mira, K. V ougioukas, P. Ma, S. Petridis, B. W. Schuller, and M. Pantic, “End-to-end video-to-speech synthesis using gener- ative adversarial networks,”IEEE transactions on cybernetics, vol. 53, no. 6, pp. 3454–3466, 2022

    2022

  5. [12]

    Lipvoicer: Generating speech from silent videos guided by lip reading,

    Y . Yemini, A. Shamsian, L. Bracha, S. Gannot, and E. Fetaya, “Lipvoicer: Generating speech from silent videos guided by lip reading,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=ZZCPSC5OgD

    2024

  6. [13]

    Learning individual speaking styles for accurate lip to speech synthesis,

    K. R. Prajwal, R. Mukhopadhyay, V . P. Namboodiri, and C. Jawa- har, “Learning individual speaking styles for accurate lip to speech synthesis,” in2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 13 793–13 802

    2020

  7. [14]

    Revise: Self-supervised speech resynthesis with visual input for univer- sal and generalized speech regeneration,

    W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y . Adi, “Revise: Self-supervised speech resynthesis with visual input for univer- sal and generalized speech regeneration,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2023, pp. 18 795–18 805

    2023

  8. [15]

    Intelligible lip-to-speech synthe- sis with speech units,

    J. Choi, M. Kim, and Y . M. Ro, “Intelligible lip-to-speech synthe- sis with speech units,” inInterspeech 2023, 2023, pp. 4349–4353

    2023

  9. [16]

    Uni-dubbing: Zero-shot speech synthe- sis from visual articulation,

    S. Lei, X. Cheng, M. Lyu, J. Hu, J. Tan, R. Liu, L. Xiong, T. Jin, X. Li, and Z. Zhao, “Uni-dubbing: Zero-shot speech synthe- sis from visual articulation,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), 202...

    2024

  10. [17]

    Lip-to-speech synthesis in the wild with multi-task learning,

    M. Kim, J. Hong, and Y . M. Ro, “Lip-to-speech synthesis in the wild with multi-task learning,” inICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  11. [18]

    MultiVerse: Efficient and expressive zero-shot multi-task text-to-speech,

    T. Bak, Y . Eom, S. Choi, and Y .-S. Joo, “MultiVerse: Efficient and expressive zero-shot multi-task text-to-speech,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for ...

    2024

  12. [19]

    Towards accurate lip-to-speech synthesis in-the-wild,

    S. Hegde, R. Mukhopadhyay, C. Jawahar, and V . Namboodiri, “Towards accurate lip-to-speech synthesis in-the-wild,” inPro- ceedings of the 31st ACM International Conference on Multime- dia, 2023, pp. 5523–5531

    2023

  13. [20]

    DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embed- ding ,

    J. Choi, J. Hong, and Y . M. Ro, “ DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embed- ding ,” in2023 IEEE/CVF International Conference on Computer Vision (ICCV). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2023, pp. 7778–7787. [Online]....

    2023

  14. [21]

    The source–filter theory of speech,

    I. Tokuda, “The source–filter theory of speech,” inOxford Re- search Encyclopedia of Linguistics, 2021

    2021

  15. [23]

    Learning pronunciation from a foreign language in speech synthesis networks,

    Y . Lee, S. Shon, and T. Kim, “Learning pronunciation from a foreign language in speech synthesis networks,” 2020. [Online]. Available: https://arxiv.org/abs/1811.09364

    2020 arXiv

  16. [24]

    Fastpitchfor- mant: Source-filter based decomposed modeling for speech syn- thesis,

    T. Bak, J.-S. Bae, H. Bae, Y .-I. Kim, and H.-Y . Cho, “Fastpitchfor- mant: Source-filter based decomposed modeling for speech syn- thesis,” inInterspeech 2021, 2021, pp. 116–120

    2021

  17. [25]

    Deep audio-visual speech recognition,

    T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisser- man, “Deep audio-visual speech recognition,”IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 12, pp. 8717–8727, 2018

    2018

  18. [26]

    Clova baseline system for the voxceleb speaker recognition challenge 2020,

    H. S. Heo, B.-J. Lee, J. Huh, and J. S. Chung, “Clova baseline system for the voxceleb speaker recognition challenge 2020,”

    2020

  19. [27]

    Svts: Scalable video-to-speech synthe- sis,

    R. Schoburg Carrillo de Mira, A. Haliassos, S. Petridis, B. W. Schuller, and M. Pantic, “Svts: Scalable video-to-speech synthe- sis,” inInterspeech 2022, 2022, pp. 1836–1840

    2022

  20. [28]

    SECS scores are computed viaResemblyzer2, while WER is derived from Auto-A VSR [22]

    for naturalness, SECS for speaker similarity, and WER for intelligibility. SECS scores are computed viaResemblyzer2, while WER is derived from Auto-A VSR [22]. We further intro- duce MAEF0 and MAERMSE to quantify prosodic discrepancies and RTF to measure system speed in genera...

  21. [29]

    Fastspeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,”

  22. [30]

    Learning audio-visual speech representation by masked multimodal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” 2022. [Online]. Available: https://arxiv.org/abs/2201. 02184

    2022

  23. [31]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-avsr: Audio-visual speech recognition with automatic labels,” inICASSP 2023 - 2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  24. [33]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech 2020, 2020, pp. 5036–5040

    2020

  25. [35]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” 2018. [Online]. Available: https://arxiv.org/abs/1809.00496

    2018 arXiv

  26. [37]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,” inInterspeech 2022, 2022, pp. 4521–4525

    2022

  27. [38]

    An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,

    J. Jensen, C. H. Taal, J. Jensen, and C. H. Taal, “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,”IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 24, no. 11, p. 2009–2022, Nov. 2016. [Online]. Available: https://doi.org/10.1109...

    2009

  28. [39]

    V2c: Visual voice cloning,

    Q. Chen, M. Tan, Y . Qi, J. Zhou, Y . Li, and Q. Wu, “V2c: Visual voice cloning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 21 242–21 251

    2022

  29. [40]

    Mediapipe: A framework for building perception pipelines,

    C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee, W.-T. Chang, W. Hua, M. Georg, and M. Grundmann, “Mediapipe: A framework for building perception pipelines,” 2019. [Online]. Available: https://arxiv.org/abs/1906.08172

    2019 arXiv

  30. [1960]

    Available: https://www.amazon.com/ Acoustic-Production-Description-Analysis-Contemporary/dp/ 9027916004

    [Online]. Available: https://www.amazon.com/ Acoustic-Production-Description-Analysis-Contemporary/dp/ 9027916004

  31. [2020]

    Available: https://arxiv.org/abs/2009.14153

    [Online]. Available: https://arxiv.org/abs/2009.14153

    2009 arXiv

  32. [2022]

    Available: https://arxiv.org/abs/2006.04558

    [Online]. Available: https://arxiv.org/abs/2006.04558

    2006 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.