Pith. sign in

REVIEW 4 major objections 6 minor 33 references

Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that replacing the conventional per-frame reconstruction loss with a phonetically weighted loss improves speech-driven 3D facial animation across four baseline models and four datasets.

desk verdict A simple loss swap with consistent small gains, but the phonetic-context claim is unsupported and the key experiment is confounded by rescaling. read the letter →

arxiv 2507.20568 v2 pith:F3QVW3F3 submitted 2025-07-28 cs.CV cs.AI

Hyung Kyu Kim , Hak Gu Kim This is my paper · ORCID
classification cs.CVcs.AI
keywords speech-driven3Dfacialanimationvisemecoarticulationphoneticcontext-awarelosstemporalweightingreconstructionVOCASETlipsynchronization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard per-frame reconstruction loss used to train speech-driven 3D facial animation models is a cause of jittery, unnatural lips because it treats every frame equally and ignores coarticulation. It proposes replacing that loss with a phonetic context-aware loss that weights each frame by how much the facial vertices actually move inside a short temporal window, so training concentrates on the transitions between visemes. If the claim is right, any existing animator that uses a reconstruction loss can be improved, without new network architecture or extra data, simply by swapping the loss function. The paper reports consistent gains in face and lip error metrics and smoother visual transitions across four recent models and four datasets.

What carries the argument

The load-bearing object is the viseme coarticulation weight $\tilde{w}_t$, a softmax-normalized measure of local articulatory motion. It is computed from ground-truth vertex trajectories alone: $w_t = (1/|\Omega_t^\sigma|) \sum_{k\in\Omega_t^\sigma} \|v_k - v_{k-1}\|^2$ with window radius $\sigma=2$, then normalized across the sequence to form $\tilde{w}_t$. Multiplied frame-by-frame into the reconstruction loss, it does the work of telling the network which instants matter most: moments of large, fast lip and cheek movement get emphasized, and quiet frames are down-weighted, which the paper argues encodes the gradual, context-driven emergence of visemes rather than abrupt frame-wise targets.

What would settle it

A direct check: take a trained baseline and replace $\tilde{w}_t$ with either (a) frame-to-frame speed $\|v_t - v_{t-1}\|^2$ normalized the same way, or (b) the same weights but randomly permuted across time. Train and compare FVE and LDTW on VOCASET. If (a) matches $\mathcal{L}_{pc}$'s gains, the five-frame temporal context is doing nothing beyond speed; if (b) also matches, the weights are just a per-frame gate and the phonetic-context claim is unsupported. A complementary test is to correlate $w_t$ with phoneme-boundary annotations: high weight should be concentrated at phoneme transitions.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that explicitly modeling phonetic context-dependent visemes through the loss function is what improves speech-driven 3D facial animation. The authors define a viseme coarticulation weight $\tilde{w}_t = \exp(w_t) / \sum_i \exp(w_i)$, where $w_t$ is the average squared displacement of all facial vertices over a symmetric window of five frames around time $t$, and use it to reweight the per-frame reconstruction error: $\mathcal{L}_{pc} = \sum_t \tilde{w}_t \|v_t - \hat{v}_t\|^2$. Frames in which the mouth is moving rapidly during a phoneme-to-phoneme transition receive higher weight, and frames that are nearly static receive less. Replacing $\mathcal{L}_{rec}$ with $\mathcal{L}_{pc}$ in four existing animators, FaceFormer, CodeTalker, SelfTalk, and ScanTalk, is reported to lower FVE, LVE, LDTW, and Lip-max on VOCASET, BIWI, BIWI6, and MultiFace, with the largest reductions on LVE and LDTW, and the authors interpret this as evidence that context-dependent viseme transitions were being learned rather than ignored.

Load-bearing premise

The method stands or falls on whether $w_t$, the average squared vertex displacement inside a five-frame window, really measures phonetic-context-dependent viseme transitions rather than just how fast the whole face is moving; if it only tracks general motion, the loss is a saliency reweighting and the paper's phonetic story collapses.

Editorial extensions

If this is right

  • Four different architectures, a transformer, a VQ-VAE, a self-supervised diagram model, and a diffusion model, all improve when only the reconstruction loss is replaced, suggesting the loss is architecture-agnostic.
  • The smaller lip-region errors (LVE, LDTW, Lip-max) on VOCASET and BIWI imply that the benefit is concentrated precisely where coarticulation is visible, in the lips.
  • The ablation over window sizes shows the improvement holds for every tested window, with window size 5 giving the lowest FVE and LVE, so the method has a tunable but robust temporal context.
  • Because the loss uses only vertex positions and audio-aligned targets, it can be dropped into any supervised 3D facial animator that already has per-frame vertex supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The phonetic interpretation is not directly tested: $w_t$ is computed from geometry, not from phoneme labels, so an equally consistent reading is that $\mathcal{L}_{pc}$ is a motion-saliency weighting; distinguishing the two would require correlating $w_t$ with annotated phoneme boundaries or comparing against a weight based on frame-to-frame speed alone.
  • Nothing in the losses prevents combining $\mathcal{L}_{pc}$ with a temporally weighted velocity loss; if the weighting is truly phonetic, a velocity term scaled by the same $\tilde{w}_t$ might remove residual jitter in the fast-transition frames that are now emphasized.
  • The same weighted-reconstruction idea should transfer to other articulated targets with vertex-level supervision, such as body motion or hand tracking, where coarticulation-like anticipation effects exist; the paper does not test this.
  • A stronger test of the phonetic claim would be to train on the same data while shuffling the temporal ordering of the weights; if the results stay the same, only the per-frame magnitudes matter, not the temporal context.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes replacing the conventional per-frame reconstruction loss with a "phonetic context-aware loss" L_pc in speech-driven 3D facial animation models. The loss weights each frame's vertex error by a softmax-normalized average squared vertex displacement over a symmetric temporal window (Eqs. 4-6), intending to emphasize frames where articulators move more due to coarticulation. Experiments replace L_rec with L_pc in FaceFormer, CodeTalker, SelfTalk, and ScanTalk and report improved FVE, LVE, LDTW, and Lip-max on VOCASET, BIWI, BIWI6, and MultiFace, along with a window-size ablation.

Significance. If the empirical claim holds, the paper offers a simple, easily adoptable training loss that improves several existing strong baselines across multiple datasets without architectural changes. The consistent direction of improvement across models and datasets is a positive sign, and the paper is clearly written. However, the current evidence does not establish the specifically phonetic mechanism: the loss is a motion-saliency weighting, and the reported gains may be partly or wholly attributable to the rescaling of the reconstruction term relative to other losses. Once the confounding rescaling is addressed with a uniform-weight control, the loss could still be a useful practical contribution even if the phonetic interpretation is weakened.

major comments (4)
  1. [§2.2, Eqs. (1) and (6)] The comparison between L_pc and L_rec conflates per-frame weighting with a global rescaling of the reconstruction objective. In Eq. (1), L_rec = Σ_t ||v_t − v̂_t||² is a sum over T frames, while Eq. (6) defines L_pc = Σ_t w̃_t ||v_t − v̂_t||² with Σ_t w̃_t = 1, so L_pc is a weighted average. Replacing L_rec with L_pc therefore multiplies the reconstruction term by roughly 1/T relative to L_vel in Eq. (2) and any other loss terms, in addition to redistributing weight across frames. The window-size ablation in Fig. 4 keeps the sum-to-one normalization fixed and cannot separate these two effects. A control experiment with uniform weights (e.g., L_rec/T) or with the same softmax-weighted scheme applied to a non-phonetic saliency signal is needed to support the claim that the viseme coarticulation weights, rather than the rescaling, drive the reported improvements.
  2. [§2.2, Eq. (4)] The quantity w_t defined in Eq. (4) is the average squared displacement of all facial vertices over a symmetric temporal window. The manuscript asserts that this measures "the extent of phonetic context-dependent viseme" but provides no link between w_t and phonetic units: no phoneme boundaries, articulator-specific weighting, or analysis correlating w_t with known coarticulation patterns. As written, w_t primarily rewards frames with large overall motion, which could be fast global head movement, blinks, or noise rather than phonetically meaningful articulation. To support the central claim, the authors should either show that w_t aligns with phonetic context (e.g., peaks at phoneme transitions and varies with adjacent phones) or soften the claim to "motion-saliency weighting."
  3. [Table 1 and §3.1.3] Quantitative results are reported as single point estimates without error bars, significance tests, or multiple seeds. Many differences are small — for example, FaceFormer FVE on VOCASET changes from 0.637 to 0.633 and BIWI LDTW is 0.140 in both settings — so it is unclear whether the improvements are statistically reliable. The paper's cross-model and cross-dataset claim would be substantially strengthened by reporting mean and standard deviation over at least three training runs and, where feasible, paired significance tests.
  4. [§2.2 and Fig. 4] The window radius σ=2 is fixed to a 5-frame window based on VOCASET, but the four datasets have different frame rates (60 fps for VOCASET, 25 fps for BIWI and BIWI6, 30 fps for MultiFace). A 5-frame window therefore corresponds to temporal intervals of about 80 ms on VOCASET, 160 ms on BIWI, and 133 ms on MultiFace, meaning the effective phonetic context length differs across datasets. The authors should either tune σ per dataset or justify why a fixed frame count is appropriate.
minor comments (6)
  1. [§2.2, Eq. (4)] The notation ||·||² in Eq. (4) is ambiguous; the double bars likely denote the L2 norm over vertices, which should be stated explicitly in the text.
  2. [Abstract and Conclusion] The abstract and conclusion repeatedly say the method "explicitly models" phonetic context, but no phoneme-level supervision or phonetic representation is used in the loss; this phrasing should be revised to avoid overclaiming.
  3. [Fig. 4] The legend and axis labels in Fig. 4 are difficult to read, and the caption should state which baselines are averaged for the dotted and solid lines.
  4. [§2.2, after Eq. (6)] The sentence following Eq. (6) is missing a final period, and the equation display is missing the closing period; please fix these minor typographical issues.
  5. [§6, References] Reference [21] (MultiFace) lists many authors with an incomplete author list; the full list or "et al." should be used consistently. Also, if this is a journal submission rather than an INTERSPEECH paper, the citation format should be adjusted accordingly.
  6. [Fig. 3] The caption of Fig. 3 says "Mean Std" and "Vertex Displacement" but does not define the scaling (e.g., ×10^5 units); please clarify the units and what the mean and standard deviation are computed over.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the loss is an empirical training objective evaluated on held-out benchmarks; the one self-citation is background only, and the loss-scale change is a potential confound, not circularity.

full rationale

The central claim is empirical: replacing L_rec with L_pc improves FVE, LVE, LDTW, and Lip-max across VOCASET, BIWI, BIWI6, and MultiFace. L_pc (Eq. 6) is defined from ground-truth vertex displacements via Eqs. (3)-(5); it is a training objective, not a fitted predictor used at inference. Evaluation follows the official held-out splits of each baseline model, and the window size sigma=2 is selected by the VOCASET ablation in Fig. 4 before being applied to other datasets. No equation reduces the reported improvements to the loss definition by construction, and no fitted parameter is renamed as a prediction. The only self-citation (Ref. [3], the authors' prior ICIP work) appears in a background citation list '[1, 2, 3, 4]' for immersive applications and is not load-bearing for the method or conclusion. The skeptic's observation that L_pc is a softmax-weighted average while L_rec is a sum over T (Eqs. (1) vs (6)) identifies a possible experimental confound about loss rescaling versus per-frame weighting, but a confound is an experimental-design concern, not circularity: the loss definition does not guarantee the reported quantitative gains, which are obtained from independent held-out evaluation. The paper therefore shows no significant circularity; score 1 reflects only the presence of a non-load-bearing self-citation.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The ledger is small: one hyperparameter, the window size, and three assumptions about the relationship between vertex displacement and coarticulation. No physical entities are introduced. The central result depends on the first assumption; without it, the method is just motion-saliency weighting.

free parameters (1)
  • Temporal window radius sigma (window size 2*sigma+1) = sigma = 2 (window size 5)
    Chosen in Sec. 3.4 based on lowest FVE and LVE on VOCASET; no separate validation split is reported. The softmax temperature in Eq. (5) is implicitly 1 and is not treated as a free parameter.
assumptions (3)
  • ad hoc to paper Vertex displacement magnitude inside a temporal window represents phonetic context-dependent viseme transition.
    Eqs. (4) and (5) define the viseme coarticulation weight solely from ||v_k - v_{k-1}||^2; no phoneme or viseme information is used, so the mapping from articulation magnitude to phonetic context is assumed, not derived.
  • domain assumption A symmetric window of two frames on each side captures both backward and forward coarticulation.
    Eq. (3) uses k in [t-sigma, t+sigma]; the method never distinguishes preceding vs. following context, yet the motivation in Sec. 1 explicitly invokes backward and forward coarticulation.
  • domain assumption Baseline training recipes remain valid when only the reconstruction loss is replaced.
    Sec. 3.2 states that all other objective functions from the official code are unchanged; the paper does not report re-tuning of loss weights or schedules.
invented entities (1)
  • Viseme coarticulation weight tilde{w}_t
    purpose: Softmax-normalized per-frame weight that upweights reconstruction error at high-motion frames, defined in Eqs. (5) and (6).
    It is a training-time construct defined from ground-truth vertex displacements; it makes no falsifiable prediction outside the training objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation." pith.science (2026). https://pith.science/paper/F3QVW3F3

@misc{pith2026250720568,
  author       = {Pith},
  title        = {Pith review of: Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3QVW3F3}},
  note         = {Machine review of arXiv:2507.20568}
}
read the original abstract

Speech-driven 3D facial animation aims to generate realistic facial movements synchronized with audio. Traditional methods primarily minimize reconstruction loss by aligning each frame with ground-truth. However, this frame-wise approach often fails to capture the continuity of facial motion, leading to jittery and unnatural outputs due to coarticulation. To address this, we propose a novel phonetic context-aware loss, which explicitly models the influence of phonetic context on viseme transitions. By incorporating a viseme coarticulation weight, we assign adaptive importance to facial movements based on their dynamic changes over time, ensuring smoother and perceptually consistent animations. Extensive experiments demonstrate that replacing the conventional reconstruction loss with ours improves both quantitative metrics and visual quality. It highlights the importance of explicitly modeling phonetic context-dependent visemes in synthesizing natural speech-driven 3D facial animation. Project page: https://cau-irislab.github.io/interspeech25/

Figures

Figures reproduced from arXiv: 2507.20568 by the authors.

Figure 1
Figure 1. Visualization of the audio and viseme when pronounc￾ing the word "A crab" and "A calico". Note that the phoneme represents the corresponding dominant viseme. and inertia of the articulators, i.e., phonetic context-dependent viseme [7, 8]. Previous works for speech-driven 3D facial animation used temporal-specific networks such as Temporal CNN (T-CNN), LSTM, and Transformer to capture the dynamics of audio [9, 10, 11… view at source ↗
Figure 2
Figure 2. Visual comparisons of baseline models vs. baseline models trained with our objective function on VOCASET. Note that the second, fourth, and sixth rows represent the visualization of per-vertex errors in lip regions. of which are shared by all speakers. Every 3D face model has 5,023 vertices and is registered to the FLAME [25] topology. BIWI. BIWI consists of 40 unique sentences shared across all speakers. There are … view at source ↗
Figure 3
Figure 3. Qualitative comparison of optimizing the baseline models with their original objective functions vs. with our pho￾netic context-aware loss instead of reconstruction loss. ScanTalk [19] introduces a diffusion-based approach to enable facial animation synthesis for various topologies with a single model regardless of the topology between various datasets. 3.1.3. Metrics For quantitative evaluation, we employ four metr… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 30 canonical work pages

  1. [1]

    Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation

    Introduction Speech-driven 3D facial animation aims to predict realistic 3D facial deformation fields, which change a given static facial mesh template, synced with input audio. As this task is often regarded as a key generative AI technology for immersive ap- plications such as VR remote presence, filmmaking, and game character animation [1, 2, 3, 4], it...

    work page Pith review arXiv 2025

  2. [2]

    Method In this section, we illustrate the proposed method with the common objective functions used in existing speech-driven 3D facial animator baselines such as FaceFormer [16] and CodeTalker [17]. This section consists of two subsections: 1) common objective functions in speech-driven 3D facial anima- tor baselines as a preliminary and 2) learning phone...

    work page

  3. [3]

    #$%ℒ!"#$%→𝓛𝒑𝒄GTCodeTalkerℒ!

    Experiments 3.1. Experimental Settings 3.1.1. Datasets In our experiments, we conduct extensive experiments on four widely-used datasets, which are VOCASET [15], BIWI [20], BIWI6 [19, 20], and MultiFace [21]. These datasets include pairs of audio and the corresponding 3D facial scans that show the pronunciation of English speech. VOCASET.VOCASET comprises...

    work page 1950

  4. [4]

    Conclusion In this paper, we introduced a phonetic context-aware loss to en- hance the naturalness of speech-driven 3D facial animation by explicitly modeling viseme transitions influenced by phonetic context. Unlike existing approaches that treat all facial ver- tices equally in the reconstruction loss, our method leverages a viseme coarticulation weight...

    work page

  5. [5]

    Acknowledgments This research was supported by the National Research Foun- dation of Korea(NRF) grant funded by the Korea govern- ment(MSIT) (RS-2023-00253232) and Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2023 (Project Name: Acquisition of 3D precise in...

    work page 2023

  6. [6]

    Exploring phonetic context-aware lip-sync for talking face generation,

    S. J. Park, M. Kim, J. Choi, and Y . M. Ro, “Exploring phonetic context-aware lip-sync for talking face generation,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 4325–4329

    work page 2024

  7. [7]

    Synthesizing obama: learning lip sync from audio,

    S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Synthesizing obama: learning lip sync from audio,”ACM Trans- actions on Graphics (TOG), vol. 36, no. 4, pp. 1–13, 2017

    work page 2017

  8. [8]

    Deep video portraits,

    H. Kim, P. Garrido, A. Tewari, W. Xu, J. Thies, M. Niessner, P. Pérez, C. Richardt, M. Zollhöfer, and C. Theobalt, “Deep video portraits,”ACM transactions on graphics (TOG), vol. 37, no. 4, pp. 1–14, 2018

    work page 2018

Show all 33 references
  1. [9]

    Analyzing visible articulatory movements in speech production for speech-driven 3d facial an- imation,

    H. K. Kim, S. Lee, and H. G. Kim, “Analyzing visible articulatory movements in speech production for speech-driven 3d facial an- imation,” in2024 IEEE International Conference on Image Pro- cessing (ICIP). IEEE, 2024, pp. 3575–3579

    2024

  2. [10]

    Enhancing speech-driven 3d facial animation with audio-visual guidance from lip reading expert,

    H. EunGi, O. Hyun-Bin, K. Sung-Bin, C. Nivelet Etcheberry, S. Nam, J. Ju, and T.-H. Oh, “Enhancing speech-driven 3d facial animation with audio-visual guidance from lip reading expert,” in Conference of the International Speech Communication Associa- tion (INTERSPEECH), 2024, ...

    2024

  3. [11]

    Facial anima- tion based on context-dependent visemes,

    J. M. De Martino, L. P. Magalhães, and F. Violaro, “Facial anima- tion based on context-dependent visemes,”Computers & Graph- ics, vol. 30, no. 6, pp. 971–980, 2006

    2006

  4. [12]

    Unitalker: Scaling up audio-driven 3d facial animation through a unified model,

    X. Fan, J. Li, Z. Lin, W. Xiao, and L. Yang, “Unitalker: Scaling up audio-driven 3d facial animation through a unified model,” in European Conference on Computer Vision (ECCV). Springer, 2024, pp. 204–221

    2024

  5. [13]

    Lip read- ing sentences in the wild,

    J. Son Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip read- ing sentences in the wild,” inIEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2017, pp. 6447–6456

    2017

  6. [14]

    Deep audio-visual speech recognition,

    T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisser- man, “Deep audio-visual speech recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 8717–8727, 2022

    2022

  7. [15]

    when we train baselines with or without the proposed pho- netic context-aware loss according to different window sizes. In Fig. 4, the dotted line represents the average performance of original baseline models trained using their official code. In contrast, the solid line repr...

  8. [16]

    Probabilistic speech-driven 3d facial motion synthe- sis: New benchmarks methods and applications,

    K. D. Yang, A. Ranjan, J.-H. R. Chang, R. Vemulapalli, and O. Tuzel, “Probabilistic speech-driven 3d facial motion synthe- sis: New benchmarks methods and applications,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27 294–27 303

    2024

  9. [17]

    Speech-driven 3D face animation with composite and regional facial movements,

    H. Wu, S. Zhou, J. Jia, J. Xing, Q. Wen, and X. Wen, “Speech-driven 3D face animation with composite and regional facial movements,” inACM International Conference on Multimedia (ACM MM), 2023, pp. 6822–6830. [Online]. Available: https://doi.org/10.1145/3581783.3611775

    2023

  10. [18]

    Facetalk: Audio- driven motion diffusion for neural parametric head models,

    S. Aneja, J. Thies, A. Dai, and M. Nießner, “Facetalk: Audio- driven motion diffusion for neural parametric head models,” in IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2024, pp. 21 263–21 273

    2024

  11. [19]

    Imitator: Personalized speech-driven 3d facial an- imation,

    B. Thambiraja, I. Habibie, S. Aliakbarian, D. Cosker, C. Theobalt, and J. Thies, “Imitator: Personalized speech-driven 3d facial an- imation,” inIEEE International Conference on Computer Vision (ICCV), 2023, pp. 20 621–20 631

    2023

  12. [20]

    Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models,

    Z. Sun, T. Lv, S. Ye, M. Lin, J. Sheng, Y .-H. Wen, M. Yu, and Y .- j. Liu, “Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models,”ACM Transac- tions on Graphics (TOG), vol. 43, no. 4, pp. 1–9, 2024

    2024

  13. [21]

    Capture, learning, and synthesis of 3D speaking styles,

    D. Cudeiro, T. Bolkart, C. Laidlaw, A. Ranjan, and M. J. Black, “Capture, learning, and synthesis of 3D speaking styles,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10 101–10 111. [Online]. Available: http: //voca.is.tue.mpg.de/

    2019

  14. [22]

    Faceformer: Speech-driven 3d facial animation with transformers,

    Y . Fan, Z. Lin, J. Saito, W. Wang, and T. Komura, “Faceformer: Speech-driven 3d facial animation with transformers,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18 770–18 780

    2022

  15. [23]

    Codetalker: Speech-driven 3d facial animation with discrete mo- tion prior,

    J. Xing, M. Xia, Y . Zhang, X. Cun, J. Wang, and T.-T. Wong, “Codetalker: Speech-driven 3d facial animation with discrete mo- tion prior,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 12 780–12 790

    2023

  16. [24]

    Selftalk: A self-supervised commutative training dia- gram to comprehend 3d talking faces,

    Z. Peng, Y . Luo, Y . Shi, H. Xu, X. Zhu, H. Liu, J. He, and Z. Fan, “Selftalk: A self-supervised commutative training dia- gram to comprehend 3d talking faces,” inACM International Con- ference on Multimedia (MM), 2023, pp. 5292–5301

    2023

  17. [25]

    Scantalk: 3d talking heads from unregistered scans,

    F. Nocentini, T. Besnier, C. Ferrari, S. Arguillere, S. Berretti, and M. Daoudi, “Scantalk: 3d talking heads from unregistered scans,” inEuropean Conference on Computer Vision (ECCV). Springer, 2024, pp. 19–36

    2024

  18. [26]

    A 3-d audio-visual corpus of affective communication,

    G. Fanelli, J. Gall, H. Romsdorfer, T. Weise, and L. Van Gool, “A 3-d audio-visual corpus of affective communication,”IEEE Trans- actions on Multimedia, vol. 12, no. 6, pp. 591–598, 2010

    2010

  19. [27]

    Multiface: A dataset for neural face rendering,

    C.-h. Wuu, N. Zheng, S. Ardisson, R. Bali, D. Belko, E. Brockmeyer, L. Evans, T. Godisart, H. Ha, X. Huang, A. Hypes, T. Koska, S. Krenn, S. Lombardi, X. Luo, K. McPhail, L. Millerschoen, M. Perdoch, M. Pitts, A. Richard, J. Saragih, J. Saragih, T. Shiratori, T. Simon, M. Stew...

    2022 arXiv

  20. [28]

    Lipreading using temporal convolutional networks,

    B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” inIEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 6319–6323

    2020

  21. [29]

    Masked lip-sync prediction by audio-visual contextual exploitation in transformers,

    Y . Sun, H. Zhou, K. Wang, Q. Wu, Z. Hong, J. Liu, E. Ding, J. Wang, Z. Liu, and K. Hideki, “Masked lip-sync prediction by audio-visual contextual exploitation in transformers,” inACM Transactions on Graphics (SIGGRAPH Asia), 2022, pp. 1–9

    2022

  22. [30]

    Modeformer: Modality- preserving embedding for audio-video synchronization using transformers,

    A. Gupta, R. Tripathi, and W. Jang, “Modeformer: Modality- preserving embedding for audio-video synchronization using transformers,” inICASSP 2023 - 2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  23. [31]

    Learning a model of facial shape and expression from 4d scans,

    T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero, “Learning a model of facial shape and expression from 4d scans,”ACM Trans- actions on Graphics (SIGGRAPH), vol. 36, no. 6, 2017

    2017

  24. [32]

    Toward accurate dynamic time warping in linear time and space,

    S. Salvador and P. Chan, “Toward accurate dynamic time warping in linear time and space,”Intelligent Data Analysis, vol. 11, no. 5, pp. 561–580, 2007

    2007

  25. [33]

    Meshtalk: 3D face animation from speech using cross-modality disentanglement,

    A. Richard, M. Zollhöfer, Y . Wen, F. de la Torre, and Y . Sheikh, “Meshtalk: 3D face animation from speech using cross-modality disentanglement,” inIEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 1153–1162

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.