Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Text2Lip claims that a text-only, viseme-guided pipeline can generate talking-face videos whose lip sync and visual quality match or beat audio-driven baselines on GRID and AVDigits.

desk verdict The text-only viseme framework is a genuinely new idea, but Table 1 contradicts the paper's headline superiority claim, so the current manuscript does not support its central result. read the letter →

arxiv 2508.02362 v1 pith:VI2HF4HI submitted 2025-08-04 cs.CV cs.AI

Xu Wang , Shengeng Tang , Fei Wang , Lechao Cheng , Dan Guo , Feng Xue , Richang Hong This is my paper · ORCID
classification cs.CVcs.AI
keywords talkingfacegenerationtext-drivensynthesisvisemelipsynchronizationcurriculumlearningpseudo-audiolandmark-guidedrenderingcross-modalattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that talking-face videos can be driven by text alone, with no audio at inference, and still match or beat audio-driven systems in lip sync, visual quality, and semantic fidelity. Text2Lip maps text to IPA phonemes and then to visemes, using those viseme sequences as a linguistically grounded visual-articulation prior, and it trains with a curriculum that progressively swaps real audio for 'pseudo-audio' reconstructed from viseme features. On the GRID and AVDigits benchmarks the authors report higher BLEU and lower WER for landmark semantics, competitive or better SSIM, LPIPS, FID, and FVD, and higher user-study ratings than several audio-based baselines. The payoff would matter because high-quality lip-synced avatars would no longer require expensive, carefully aligned audio-video recordings, opening text-driven generation for low-resource and privacy-sensitive applications.

What carries the argument

The load-bearing objects are the viseme sequence and the Progressive Viseme-Audio Replacement (PVAR) curriculum. A Bernoulli mask with dropout probability $p_{\mathrm{drop}}$ rising linearly from 0 to 1 (Eq. 5) progressively removes real audio during training, forcing a Transformer-based generator to reconstruct pseudo-audio $\hat{a}_{1:N}$ from enhanced viseme features through multi-head cross-modal attention (Eq. 7); the pseudo-audio is gated and fused with viseme features, and a Transformer LipDecoder predicts landmarks that an EchoMimic-based renderer turns into video. The viseme mapping supplies the semantic disambiguation, the curriculum supplies the modality transfer, and the landmark renderer supplies the photorealistic output.

What would settle it

Recompute FVD on the GRID test set with the paper's protocol: if EchoMimic's reported FVD of 251.116 is reproducible, Text2Lip's 277.65 is not the best FVD, which would settle the strongest form of the visual-realism claim. A second check targets the core mechanism: on a held-out set of viseme-confusable minimal pairs (e.g., 'bad boy' vs 'bat boat'), generated videos should be classified by a lipreading model at above-chance accuracy; near-chance accuracy would falsify the claimed semantic disambiguation.

Watch

Extended reading notes

Core claim

The central claim is that a viseme-centric text-only pipeline outperforms audio-driven talking-face methods in semantic fidelity, visual realism, and modality robustness, and that despite relying solely on text, Text2Lip surpasses or approaches the audio-based baseline in all metrics. The authors attribute the gain to an explicit linguistic-phonetic-visual hierarchy: text becomes IPA phonemes, phonemes are grouped into visemes via the Microsoft Speech API mapping, and those visemes act as a semantically grounded prior that avoids audio-to-lip ambiguities such as the near-identical lip shapes of 'bad boy' and 'bat boat'. The method then reconstructs audio-like features from the viseme sequence and renders video from predicted landmarks, so the same architecture handles audio-present, noisy-audio, and audio-free conditions.

Load-bearing premise

The framework assumes that the phoneme-to-viseme mapping from the Microsoft Speech API plus text context preserves enough articulatory and temporal information for cross-modal attention to reconstruct useful audio-like features; if acoustically distinct phonemes such as /p/ and /b/ or /s/ and /z/ collapse into the same viseme and the model cannot recover the distinction, sentences like 'bad boy' versus 'bat boat' would remain confused.

Editorial extensions

If this is right

  • If the claim holds, aligned audio-video corpora are no longer a prerequisite for lip-synced talking faces: text alone drives the full pipeline.
  • Viseme grounding should make generation robust to noisy or missing audio, since the model reconstructs audio-like features from text rather than depending on the audio channel.
  • Text-level control becomes interpretable: editing the input sentence changes the viseme sequence and therefore the predicted lip motion in a structured way.
  • On GRID and AVDigits, text-only generation can match or exceed audio-driven baselines across semantic and visual metrics, including under modality-incomplete conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to quantify information loss in the phoneme-to-viseme compression: compute how well pseudo-audio MFCCs reconstructed from visemes predict the original MFCCs, or how well a TTS built on the pseudo-audio preserves prosody and stress.
  • Because GRID's vocabulary is only 51 words, the reported gains may be larger than what open-domain vocabulary would give; a natural stress test is to repeat the landmark-semantics evaluation on a large-vocabulary or unconstrained speech corpus.
  • The progressive dropout schedule suggests a general recipe for modality-agnostic training: any pair of input modalities could be swapped along a curriculum that ends with the cheaper modality alone, not just real audio to pseudo-audio.
  • If homophones and viseme-identical words are systematically confused in the generated lip motion, pairing the viseme prior with a language model at the text stage could restore the distinction without returning to audio input.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Text2Lip, a text-driven talking face generation framework that converts text into viseme sequences via a word-to-phoneme-to-viseme pipeline, trains a Transformer with a progressive viseme-audio replacement curriculum to reconstruct pseudo-audio features, and renders videos with a landmark-guided EchoMimic backend. The authors claim state-of-the-art performance in semantic fidelity, visual realism, and modality robustness on GRID and AVDigits, and support this with quantitative comparisons, ablations, and a user study. The central claim is that a text-only viseme-driven model can match or beat audio-driven baselines on these benchmarks.

Significance. If the claims were supported, a text-only viseme-guided pipeline that matches audio-driven talking face generation would be a meaningful step for low-resource and privacy-sensitive applications, and the interpretable viseme intermediate representation is a plausible and interesting design. The paper is clearly written in terms of architecture, and the curriculum-based replacement and pseudo-audio reconstruction are well-motivated. However, the manuscript does not ship code, data splits, or error bars, and the central quantitative claims are internally inconsistent with Table 1, so the significance cannot be assessed from the submitted evidence.

major comments (4)
  1. [Comparison with State-of-the-Arts, Table 1] The prose and Table 1 contradict each other on the headline FVD metric. On GRID, the text claims Text2Lip 'achieves the best FVD score of 277.65, outperforming all compared methods,' but Table 1 lists EchoMimic with FVD 251.116, which is better (lower). On AVDigits, the text says Text2Lip's FVD 251.841 surpasses V-Express (176.67), but 176.67 is V-Express's FID value in the table; V-Express's FVD is 562.891. The text also attributes FVD 451.286 to SadTalker, whereas Table 1 shows 451.286 as AniPortrait's FVD, with SadTalker's FVD at 308.519, and Sonic's AVDigits FVD is 266.924, not 308.594. The table shows Text2Lip's AVDigits FVD as 251.441, not 251.841. Because FVD is one of the two headline video-quality metrics, these inconsistencies directly undermine the central 'outperforms existing approaches' claim. The absence of code, data splits, and error bars leaves the reader unable to determine which numbers are correct.
  2. [Evaluation Metrics and Table 2] The semantic fidelity comparison is not established as a fair or independent test. The paper reports BLEU, WER, and DTW-P from a retrained NSLT model, but does not describe the NSLT training protocol, the test vocabulary, or whether the identical protocol is applied to all baselines. Since Text2Lip is conditioned on the exact text whose recoverability is being measured, high semantic scores may reflect that the text is available as an input rather than that the articulation is semantically accurate. Moreover, the landmark baselines in Table 2 (PT-GN, GEN-OBT, LVMCN) are sign language production models, not talking face generation systems, so the comparison does not support the claimed semantic-fidelity advantage over talking face methods.
  3. [Viseme-Centric Text Encoding and Motivating Example] The motivating failure mode is that 'bad boy' and 'bat boat' produce nearly identical lip motions, but the proposed phoneme-to-viseme mapping explicitly collapses acoustically distinct phonemes such as /b/ with /p/ and /s/ with /z/. The manuscript provides no experiment showing that the viseme representation, after pseudo-audio reconstruction, actually disambiguates such minimal pairs. Without a minimal-pair analysis or an information-theoretic characterization of the viseme compression, the claim of improved semantic precision over audio-driven methods remains unsupported.
  4. [Training Protocol and Modality Robustness Claims] The claim of modality robustness is not quantitatively demonstrated. No experiment varies audio quality, noise level, or missing audio for the proposed model; Table 3 compares pseudo-audio sources rather than the model's behavior under corrupted or absent audio. The curriculum schedule in Eq. (5) is set with p_start=0 and p_end=1, but the total number of training steps T is never reported, and there is no description of how inference differs when real audio is available versus absent. These omissions make the robustness claim qualitative rather than evidence-based.
minor comments (6)
  1. [Table 1] Several entries in Table 1 are concatenated without separators (for example, '3.1985.742', '458.2575.851', and '266.9243.892'), which makes the table difficult to read and likely contributed to the prose/table mismatches.
  2. [Table 3 and Table 4] Table 3 and Table 4 label BLEU-4 with a down arrow, but higher BLEU-4 is better; the arrow should be an upward arrow. Table 4's ablations text says 'LPIPS increases by 18.5% to 0.238,' but LPIPS is lower-better and the value decreases from 0.292 to 0.238.
  3. [Comparison with State-of-the-Arts] The sentence 'its MPJPE 783.32 performance is lower than GEN-OBT' is ambiguous because higher MPJPE is worse; the intended meaning appears to be 'worse than' rather than 'lower than.'
  4. [Figure 3] The caption contains a typo: 'radually' should be 'gradually.'
  5. [Experimental Settings / Evaluation Metrics] The evaluation section lists ROUGE as a metric, but ROUGE is never reported in any table; either report it or remove it from the metric list.
  6. [User Study] The user study reports a '37% improvement in smoothness' but provides no standard deviations, no confidence intervals, and no significance test, so the strength of this claim is unclear.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported lip-sync superiority reduces to self-consistency: in text-only mode the SyncNet audio reference is the model's own pseudo-audio, which also drives the generated lips.

  1. self definitional [Methodology (Progressive Viseme-Audio Replacement, Eqs. 7-10; Photorealistic Landmark Rendering, Eq. 11); Experiments (Evaluation Metrics; Comparison on GRID)]
    "These enhanced features are fed into a cross-modal pseudo-audio generator based on multi-head attention: ˆa1:N = MultiheadAttention(˜a1:N , ˜venh 1:N , ˜venh 1:N ). ... ˜lm+1 = LipDecoder(l′1:m, ˜v1:N , ˆapse 1:N ) ... Video = Synthesis({ ˜lm+1}M m=1, audio), where audio can refer to original audio or reconstructed pseudo-audio, depending on availability. ... Sync-C measures the synchronization consistency between lip sounds and audio."

    In the claimed audio-free deployment, the only audio signal is the pseudo-audio reconstructed from viseme features (Eqs. 7-8), and the landmark sequence is predicted by cross-attending to that same pseudo-audio (Eqs. 9-10). The generated video is rendered with that pseudo-audio (Eq. 11), and SyncNet then scores synchronization between the video and 'audio' (Evaluation Metrics). The reported Sync-C/D values therefore measure whether the model's lips track the model's own pseudo-audio, not whether they sync with real speech. Comparing these scores with audio-driven baselines, which are scored against true input audio, is not commensurate; the claim of 'surpassing ... in lip synchronization' is at least partly forced by the model's internal audio loop.

full rationale

The text-to-viseme-to-landmark pipeline is an empirical training scheme, not a formal derivation, and no equation-level equivalence forces the main visual-quality results (FID/FVD/SSIM) from the inputs. The viseme mapping comes from an external API, and the renderer is adapted from EchoMimic, so no load-bearing self-citation chain was found. The one significant circularity is the lip-sync evaluation: in the text-only mode that the paper advertises, the audio reference for SyncNet is the model's own pseudo-audio, which is also a conditioning signal for landmark prediction, so the sync metric reduces to a self-consistency check rather than an independent test against real speech. This affects the central claim that Text2Lip surpasses audio-based methods on lip synchronization, though other reported metrics retain independent content; hence partial circularity (6). The paper also omits any explicit loss for pseudo-audio reconstruction, which further weakens the independence of the sync loop. Table 1 contains internal numerical inconsistencies (e.g., the prose FVD comparison contradicts the table), but those are verification/correctness issues, not circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a cascade of external priors: text-to-phoneme conversion, the Microsoft Speech API phoneme-to-viseme table, the curriculum schedule, the EchoMimic renderer, and the NSLT lipreader used for evaluation. None of these is derived in the paper or validated independently here. The pseudo-audio module is a learned mapping with no external falsifiable handle. The paper reports no sensitivity analysis for the curriculum endpoints or the viseme vocabulary.

free parameters (2)
  • Curriculum dropout endpoints (p_start=0, p_end=1) = p_start=0, p_end=1, linear ramp over T steps
    Eq. 5: chosen by hand to force full transition to text-only input; no sensitivity analysis or ablation of alternative schedules is reported.
  • Transformer and training hyperparameters = embedding 512, 2 layers, 4 heads, learning rate 1e-3, batch 128
    Reported in Model Settings as fixed choices; no ablations, but they define the model being evaluated.
assumptions (5)
  • domain assumption Text-to-phoneme conversion tools with dictionary fallback produce accurate IPA for the input text.
    Invoked in Viseme-Centric Text Encoding; conversion errors propagate directly into the viseme sequence and pseudo-audio.
  • domain assumption The Microsoft Speech API phoneme-to-viseme mapping is a valid visual articulation prior for the target datasets.
    Used as a predefined table in Viseme-Centric Text Encoding; no validation that the grouping preserves visually distinguishable distinctions.
  • domain assumption The curriculum audio dropout schedule (Eq. 5) converges to a functional text-only model.
    p_drop is linearly increased to 1, but no theoretical or empirical guarantee is given that the model does not collapse without audio.
  • domain assumption The EchoMimic renderer (Chen et al. 2025) can be adapted to synthesized landmarks and pseudo-audio without degrading quality.
    Stage III relies on an external renderer; the paper gives no analysis of the distribution shift from real to predicted landmarks and audio.
  • domain assumption The retrained NSLT lipreading model measures semantic fidelity reliably on generated videos.
    Used in Evaluation Metrics for BLEU and WER; if the lipreader exploits artifacts or the text prior, the semantic advantage is overstated.
invented entities (1)
  • Pseudo-audio feature sequence
    purpose: Substitute for real audio in landmark prediction and rendering when audio is absent at inference.
    Defined entirely by learned mappings in Eqs. 7-8; it has no falsifiable handle outside the paper's own evaluations, so it is an internal artifact rather than a validated entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering." pith.science (2026). https://pith.science/paper/VI2HF4HI

@misc{pith2026250802362,
  author       = {Pith},
  title        = {Pith review of: Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VI2HF4HI}},
  note         = {Machine review of arXiv:2508.02362}
}
read the original abstract

Generating semantically coherent and visually accurate talking faces requires bridging the gap between linguistic meaning and facial articulation. Although audio-driven methods remain prevalent, their reliance on high-quality paired audio visual data and the inherent ambiguity in mapping acoustics to lip motion pose significant challenges in terms of scalability and robustness. To address these issues, we propose Text2Lip, a viseme-centric framework that constructs an interpretable phonetic-visual bridge by embedding textual input into structured viseme sequences. These mid-level units serve as a linguistically grounded prior for lip motion prediction. Furthermore, we design a progressive viseme-audio replacement strategy based on curriculum learning, enabling the model to gradually transition from real audio to pseudo-audio reconstructed from enhanced viseme features via cross-modal attention. This allows for robust generation in both audio-present and audio-free scenarios. Finally, a landmark-guided renderer synthesizes photorealistic facial videos with accurate lip synchronization. Extensive evaluations show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness, establishing a new paradigm for controllable and flexible talking face generation. Our project homepage is https://plyon1.github.io/Text2Lip/.

Figures

Figures reproduced from arXiv: 2508.02362 by the authors.

Figure 1
Figure 1. Modality usage in talking face generation. (a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The correspondence between Viseme lip shapes [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed framework Text2Lip. It contains three key stages: (1) Viseme-Centric Text Encoding converts text to visemes via word-phoneme-viseme mapping, (2) Progressive Viseme-Audio Replacement adopts text-derived viseme embeddings to radually replace the audio using curriculum learning, and (3) Photorealistic Landmark Rendering syn￾thesizes high-quality videos with precise lip synchronization via an ad… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization examples on GRID dataset. with text embeddings during training through curriculum learning, and collaborates with the pseudo-audio generation module to re-encode the audio features. The new method uses acoustic constraints to achieve a performance leap in…
Figure 6
Figure 6. Figure 6: Visualization examples of unseen speakers in open [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A diffusion-based talking-face generator uses 3D blendshape coefficients to continuously control the emotion intensity of generated facial expressions.

Reference graph

Works this paper leans on

48 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    M.; Shillingford, B.; Whiteson, S.; and De Freitas, N

    Assael, Y. M.; Shillingford, B.; Whiteson, S.; and De Freitas, N. 2016. Lipnet: End-to-end sentence-level lipreading. arXiv preprint arXiv:1611.01599

    arXiv 2016

  4. [4]

    C.; Hadfield, S.; Koller, O.; Ney, H.; and Bowden, R

    Camgoz, N. C.; Hadfield, S.; Koller, O.; Ney, H.; and Bowden, R. 2018. Neural Sign Language Translation. In Computer Vision and Pattern Recognition, 7784--7793

    work page 2018

  5. [5]

    Chan, M. T. 2001. HMM-based audio-visual speech recognition integrating geometric-and appearance-based visual features. In IEEE Fourth Workshop on Multimedia Signal Processing, 9--14

    work page 2001

  6. [6]

    Chen, Z.; Cao, J.; Chen, Z.; Li, Y.; and Ma, C. 2025. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2403--2410

    work page 2025

  7. [7]

    S.; and Zisserman, A

    Chung, J. S.; and Zisserman, A. 2016. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV

    work page 2016

  8. [8]

    Cooke, M.; Barker, J.; Cunningham, S.; and Shao, X. 2006. An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America, 120(5): 2421--2424

    work page 2006

Show all 48 references
  1. [9]

    Cui, J.; Li, H.; Yao, Y.; Zhu, H.; Shang, H.; Cheng, K.; Zhou, H.; Zhu, S.; and Wang, J. 2025 a . Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation. In The Thirteenth International Conference on Learning Representations

    2025

  2. [10]

    Cui, J.; Li, H.; Zhan, Y.; Shang, H.; Cheng, K.; Ma, Y.; Mu, S.; Zhou, H.; Wang, J.; and Zhu, S. 2025 b . Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2...

    2025

  3. [11]

    Ferdowsifard, K.; Barke, S.; Peleg, H.; Lerner, S.; and Polikarpova, N. 2021. LooPy: interactive program synthesis with control structures. Proceedings of the ACM on Programming Languages, 5(OOPSLA): 1--29

    2021

  4. [12]

    Guo, D.; Tang, S.; and Wang, M. 2019. Connectionist Temporal Modeling of Video and Language: A Joint Model for Translation and Sign Labeling. In International Joint Conference on Artificial Intelligence, 751--757

    2019

  5. [13]

    Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30

    2017

  6. [14]

    Hore, A.; and Ziou, D. 2010. Image quality metrics: PSNR vs. SSIM. In 2010 20th international conference on pattern recognition, 2366--2369. IEEE

    2010

  7. [15]

    Hu, D.; Li, X.; et al. 2016. Temporal multimodal learning in audiovisual speech recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3574--3582

    2016

  8. [16]

    Ionescu, C.; Papava, D.; Olaru, V.; and Sminchisescu, C. 2013. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7): 1325--1339

    2013

  9. [17]

    Ji, X.; Hu, X.; Xu, Z.; Zhu, J.; Lin, C.; He, Q.; Zhang, J.; Luo, D.; Chen, Y.; Lin, Q.; et al. 2025. Sonic: Shifting focus to global audio perception in portrait animation. In Proceedings of the Computer Vision and Pattern Recognition Conference, 193--203

    2025

  10. [18]

    Ji, X.; Zhou, H.; Wang, K.; Wu, Q.; Wu, W.; Xu, F.; and Cao, X. 2022. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings, 1--10

    2022

  11. [19]

    Kazemi, V.; and Sullivan, J. 2014. One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1867--1874

    2014

  12. [20]

    Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74--81

    2004

  13. [21]

    Lucey, P.; Potamianos, G.; and Sridharan, S. 2008. Patch-based analysis of visual speech from multiple views. In International Conference on Auditory-Visual Speech Processing, 69--74

    2008

  14. [22]

    Luettin, J.; and Thacker, N. A. 1997. Speechreading using probabilistic models. Computer vision and image understanding, 65(2): 163--178

    1997

  15. [23]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311--318

    2002

  16. [24]

    J.; Kim, M.; Hong, J.; Choi, J.; and Ro, Y

    Park, S. J.; Kim, M.; Hong, J.; Choi, J.; and Ro, Y. M. 2022. Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory. In AAAI Conference on Artificial Intelligence, 2062--2070

    2022

  17. [25]

    P.; and Jawahar, C

    Prajwal, K.; Mukhopadhyay, R.; Namboodiri, V. P.; and Jawahar, C. 2020. A lip sync expert is all you need for speech to lip generation in the wild. In ACM international conference on multimedia, 484--492

    2020

  18. [26]

    Sakoe, H.; and Chiba, S. 2003. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing, 26(1): 43--49

    2003

  19. [27]

    C.; and Bowden, R

    Saunders, B.; Camgoz, N. C.; and Bowden, R. 2020. Progressive Transformers for End-to-End Sign Language Production. In European Conference on Computer Vision, 687--705

    2020

  20. [28]

    Stafylakis, T.; and Tzimiropoulos, G. 2017. Combining residual networks with LSTMs for lipreading. arXiv preprint arXiv:1703.04105

    2017 arXiv

  21. [29]

    Tang, S.; Guo, D.; Hong, R.; and Wang, M. 2022 a . Graph-Based Multimodal Sequential Embedding for Sign Language Translation. IEEE Transactions on Multimedia, 4433--4445

    2022

  22. [30]

    Tang, S.; He, J.; Cheng, L.; Wu, J.; Guo, D.; and Hong, R. 2025 a . Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observations. In Computer Vision and Pattern Recognition Conference, 3481--3491

    2025

  23. [31]

    Tang, S.; He, J.; Guo, D.; Wei, Y.; Li, F.; and Hong, R. 2025 b . Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production. In AAAI Conference on Artificial Intelligence, 7266--7274

    2025

  24. [32]

    Tang, S.; Hong, R.; Guo, D.; and Wang, M. 2022 b . Gloss Semantic-Enhanced Network with Online Back-Translation for Sign Language Production. In ACM International Conference on Multimedia, 5630--5638

    2022

  25. [33]

    Tang, S.; Xue, F.; Wu, J.; Wang, S.; and Hong, R. 2025 c . Gloss-Driven Conditional Diffusion Models for Sign Language Production. ACM Transactions on Multimedia Computing, Communications and Applications, 1--17

    2025

  26. [34]

    Unterthiner, T.; Van Steenkiste, S.; Kurach, K.; Marinier, R.; Michalski, M.; and Gelly, S. 2018. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717

    2018 arXiv

  27. [35]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is All You Need. In Neural Information Processing Systems, 1--11

    2017

  28. [36]

    Wang, C.; Tian, K.; Zhang, J.; Guan, Y.; Luo, F.; Shen, F.; Jiang, Z.; Gu, Q.; Han, X.; and Yang, W. 2024. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406.02511

    2024 arXiv

  29. [37]

    Wang, S.; Li, L.; Ding, Y.; Fan, C.; and Yu, X. 2021. Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion. In Proceedings of the Thirtieth International Joint Conference On Artificial Intelligence, Ijcai 2021, 1098--1105. International Joint Conf...

    2021

  30. [38]

    Wang, X.; Tang, S.; Cheng, L.; Li, F.; Wang, S.; and Hong, R. 2025 a . SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generation. arXiv preprint arXiv:2506.11621

    2025 arXiv

  31. [39]

    Wang, X.; Tang, S.; Song, P.; Wang, S.; Guo, D.; and Hong, R. 2025 b . Linguistics-vision monotonic consistent network for sign language production. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5. IEEE

    2025

  32. [40]

    C.; Sheikh, H

    Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600--612

    2004

  33. [41]

    Wei, C.; Sun, B.; Ma, H.; Hou, J.; Juefei-Xu, F.; He, Z.; Dai, X.; Zhang, L.; Li, K.; Hou, T.; et al. 2025. Mocha: Towards movie-grade talking character synthesis. arXiv preprint arXiv:2503.23307

    2025 arXiv

  34. [42]

    Wei, H.; Yang, Z.; and Wang, Z. 2024. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694

    2024 arXiv

  35. [43]

    Xu, M.; Li, H.; Su, Q.; Shang, H.; Zhang, L.; Liu, C.; Wang, J.; Yao, Y.; and Zhu, S. 2024 a . Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801

    2024 arXiv

  36. [44]

    Xu, S.; Chen, G.; Guo, Y.-X.; Yang, J.; Li, C.; Zang, Z.; Zhang, Y.; Tong, X.; and Guo, B. 2024 b . Vasa-1: Lifelike audio-driven talking faces generated in real time. Advances in Neural Information Processing Systems, 37: 660--684

    2024

  37. [45]

    Zhang, C.; Wang, C.; Zhang, J.; Xu, H.; Song, G.; Xie, Y.; Luo, L.; Tian, Y.; Guo, X.; and Feng, J. 2023 a . Dream-talk: Diffusion-based realistic emotional audio-driven method for single image talking face generation. arXiv preprint arXiv:2312.13578

    2023 arXiv

  38. [46]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586--595

    2018

  39. [47]

    Zhang, W.; Cun, X.; Wang, X.; Zhang, Y.; Shen, X.; Guo, Y.; Shan, Y.; and Wang, F. 2023 b . Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF conference on computer vision and patt...

    2023

  40. [48]

    Zhou, Y.; Han, X.; Shechtman, E.; Echevarria, J.; Kalogerakis, E.; and Li, D. 2020. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics, 39(6): 1--15

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.