Pith. sign in

REVIEW 3 major objections 5 minor 38 references

PD-GS: Phoneme-Driven 3DGS for Audio-Driven Talking Heads

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PD-GS injects time-aligned phoneme tokens into a 3D Gaussian splatting talker via a learned gate, achieving the best lip landmark distance (2.66) on HDTF and reducing bilabial closure violations.

desk verdict A well-motivated phoneme-gated 3DGS talker whose central claim is undercut by a small LMD gain measured on a metric it was trained to minimize. read the letter →

arxiv 2608.05218 v1 pith:2M232OPO submitted 2026-08-05 cs.AI cs.SD

Ao Fu , Yi Zhou This is my paper · ORCID
classification cs.AIcs.SD
keywords talkingheadsynthesis3DGaussianSplattingaudio-drivenanimationphoneme-guidedarticulationlipsynchronizationgatedfusionneuralrenderinglandmarkdistance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PD-GS is a 3D Gaussian Splatting talking-head system that adds explicit, frame-aligned phoneme tokens to the usual audio-driven motion prediction. The paper's central claim is that the continuous acoustic embedding alone biases a regression model toward averaged mouth shapes, so brief articulatory events like bilabial closures get undershot and produce the 'leaky mouth' artifact. Injecting discrete phonemes through a learned gate fixes this: on the HDTF benchmark PD-GS reports the best lip landmark distance among compared baselines (LMD 2.66) and, in qualitative tests, cleaner closures and more distinct vowels. The phoneme tokens come from an offline ASR and forced-alignment pipeline, so the headline latency of 9.1 ms is model-side after those tokens are available. If the claim holds, it means a small injection of linguistic structure, rather than a larger network, is what articulation accuracy was missing.

What carries the argument

The load-bearing component is the Linguistic Fusion Module (LFM), a learned gate that decides, per channel and per frame, how much of the latent articulation feature comes from continuous audio context versus a projected phoneme embedding. The gate is computed as $\overline{g}_t = \frac{1}{D}\sum_{j=1}^D g_{t,j}$, with the fusion rule $f_{\mathrm{art},t}=(1-g_t)\odot h_t+g_t\odot \mathrm{MLP}_{\mathrm{ph}}(e_{\mathrm{ph},t})$; it is trained end-to-end with image reconstruction (L1 plus LPIPS) and a lip-landmark consistency loss. Around it sits a dual-stream design that separates high-frequency lip/jaw articulation from low-frequency expression dynamics, and an offline ASR plus forced-alignment pipeline that turns each frame into one of 40 phoneme tokens (including silence). The gate's role is to keep smooth audio-driven dynamics where audio is sufficient and boost phoneme guidance precisely where articulation-critical events such as bilabial closures and strong rounding occur.

What would settle it

Take the trained PD-GS model and drive it with audio whose phoneme alignments are deliberately corrupted—randomly shifting every phoneme boundary by 50 ms, or replacing /p/, /b/, /m/ labels with randomly chosen vowels—then measure LMD on the same test clips. If LMD stays near 2.66, phoneme labels are not what carries the gain; if it degrades toward the audio-only ablation level of 2.73 or worse, the claim depends on alignment accuracy. The same test on singing voice or strongly accented speech, where ASR and forced alignment typically falter, would settle how far the method generalizes.

Watch

Extended reading notes

Core claim

PD-GS argues that the bottleneck in photorealistic talking heads is no longer rendering quality but the physical and linguistic correctness of facial motion. It attributes the over-smoothed, sometimes unsealed lip motions of audio-only systems to a fundamental mismatch: continuous regression from HuBERT embeddings to mouth configurations averages out discrete articulatory events. The proposed remedy is a dual-stream motion generator: an expression stream predicts low-frequency prosodic deformation from audio, while an articulation stream fuses HuBERT context features with time-aligned phoneme embeddings through the Linguistic Fusion Module (LFM), whose learned gate vector $g_t=\sigma(\mathrm{MLP}_{\mathrm{gate}}([h_t,e_{\mathrm{ph},t}]))$ channel-wise interpolates between audio context and phoneme projection. The paper reports that this improves lip landmark distance to 2.66 on HDTF (from 2.71 for the strongest 3DGS baseline), reduces closure violations and phonetic ambiguity in qualitative comparisons, loses the improvement when the LFM is removed (LMD 2.73), and generalizes to a VoxCeleb2 subset with the best LMD, LPIPS, and NIQE among compared methods.

Load-bearing premise

The load-bearing premise is that the automatic ASR and forced-alignment pipeline produces correct, frame-aligned phoneme labels for whatever audio drives the avatar; if the alignment is wrong, noisy, or unavailable for out-of-domain audio such as singing or heavy accents, the gate has no trustworthy phonetic signal to inject, and the method's advantage can shrink or reverse.

Editorial extensions

If this is right

  • On HDTF, PD-GS reaches the best lip landmark distance among the compared systems (LMD 2.66), ahead of GaussianTalker's 2.71, with best LPIPS and NIQE and competitive sync (8.85).
  • Ablations show the phoneme pathway is load-bearing: removing the LFM degrades LMD to 2.73 and sync to 8.71, and replacing gated fusion with plain concatenation gives 2.70, below the full model.
  • The reported MOS study gives PD-GS the highest scores on lip-sync accuracy (4.12), visual realism (4.01), and naturalness (4.07).
  • Cross-dataset results on a VoxCeleb2 subset reproduce the pattern: best LMD (2.94), LPIPS (0.031), and NIQE (3.79) among the compared methods, suggesting the gain is not specific to one recording condition.
  • Because phoneme tokens are produced offline by ASR and forced alignment, the 110 FPS / 9.1 ms figure is model-side latency after tokens exist, not fully streaming end-to-end latency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gate is genuinely learning compatibility between acoustic context and phonetic labels, corrupted or missing alignments should show a predictable failure curve: random temporal jitter in phoneme boundaries should degrade LMD smoothly, and wrong labels for minimal pairs should produce visible articulation errors that audio-only models would not make.
  • The same gated-fusion idea could be tested with richer discrete linguistic units than phonemes—stress marks, syllable boundaries, or articulatory feature vectors—to see whether closure-level events are better encoded as phonological features shared across related phonemes rather than as atomic tokens.
  • A streaming version of the phoneme front-end (incremental ASR or a lightweight phonetic classifier) would be the natural next test: if it preserves the LMD gain while removing the offline preprocessing step, the method becomes usable for live avatars; if not, the offline alignment is part of what makes the gain real.
  • The gate visualization suggests a diagnostic tool: average gate activation by phoneme class could reveal which articulatory events the model treats as critical, and unusually high gate values on silence or non-speech frames might flag alignment errors in the training data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. PD-GS augments a person-specific 3D Gaussian Splatting talking-head model with time-aligned phoneme tokens produced by an automatic ASR and forced-alignment pipeline. A dual-stream motion generator uses a HuBERT-based expression stream and a phoneme-driven articulation stream whose outputs are combined through a learned gate (the Linguistic Fusion Module, LFM). Training uses image reconstruction plus a 2D landmark-consistency loss, and inference uses offline phoneme alignment. On HDTF the method reports the best Lip Landmark Distance (LMD) among compared baselines (2.66 vs. GaussianTalker's 2.71), with ablations showing LMD improving from 2.73 without the LFM to 2.66 with it. Additional experiments include a MOS user study, a VoxCeleb2 cross-dataset evaluation, and qualitative demonstrations of bilabial closure and vowel-distinctness behavior.

Significance. If the reported results are robust, the paper makes a useful contribution by showing that explicit discrete phonetic tokens, injected through a lightweight gated module, can improve articulation fidelity in 3DGS-based talking heads while preserving rendering quality and compact model size. The architecture is plausible and the paper includes several strengths: a cross-dataset evaluation on VoxCeleb2, a user study, gate-behavior visualization, and honest discussion of the offline nature of the phoneme-extraction pipeline. However, the central quantitative claim currently rests on small LMD margins with no error bars or significance testing, and the LMD metric appears closely aligned with the landmark loss used to train PD-GS but not the baselines. The identity coverage of the main evaluation is also narrow. These issues make the significance conditional on additional statistical and metric-validation evidence.

major comments (3)
  1. [Section 4.2, Tables 1-2, Eq. (4)] The primary evidence for the central claim is the LMD margin of 0.05 over GaussianTalker (Table 1) and 0.07 over the audio-only ablation (Table 2), yet no standard deviations, confidence intervals, or multiple-seed results are reported anywhere in the paper. Moreover, the LMD metric is not formally defined and appears to be essentially the same quantity as the training loss ℒ_landmark in Eq. (4) (L2 distance between projected 3D landmarks and 2D ground-truth landmarks), which PD-GS is explicitly trained to minimize while most baselines are not. This makes it difficult to distinguish a genuine phonetic-grounding effect from run-to-run noise or from optimization alignment with the evaluation metric. Please define LMD explicitly, report variance across at least three training runs, and add at least one articulation metric that is not part of the training objective (e.g., a separately trained lip-reading classifier or a closure/no-closure detector).
  2. [Section 4.1, Datasets; Section 4.4, Table 4] The main comparison is limited to two HDTF subjects ('Obama' and 'May'), and the cross-dataset experiment in Table 4 does not state how many subjects or clips are used or how they are split. Given that the paper claims the phoneme-to-articulation mapping generalizes across identities (Section 4.5, Fig. 5), the evaluation should include more identities and specify subject-level statistics; without this, the identity-generalization claim is not supported by measured evidence.
  3. [Section 3.5, Phoneme extraction; Section 4.2, Efficiency] The method depends on correct frame-aligned phoneme labels from an automatic ASR + forced-alignment pipeline (refs [20,27]), but the paper does not quantify the accuracy of these alignments or test sensitivity to alignment errors (e.g., on accented speech or singing). Because the gate has no trustworthy phonetic signal when alignments are wrong, the claimed advantage could disappear in out-of-domain conditions; please report alignment quality on the test utterances or add a robustness experiment with corrupted or noisy phoneme labels.
minor comments (5)
  1. [Section 4.2, Evaluation Metrics] LMD is used as the primary articulation metric but is never formally defined; please give its exact formula and state whether it is computed on the same landmarks as Eq. (4).
  2. [Section 3.5, Phoneme extraction] Please expand MFA (Montreal Forced Aligner) at first use and specify the phoneme inventory and language assumption; a vocabulary size of V=40 including silence and pause appears reasonable only for English and should be stated as such.
  3. [Section 4.2, Efficiency] The FPS/latency discussion is honest about excluding offline ASR+MFA, but please report the end-to-end latency including HuBERT feature extraction and phoneme alignment, or clearly label the comparison as model-side only so that readers can compare fairly with streaming baselines.
  4. [Section 4.3, Table 3] The MOS study reports only average scores across 18 raters and 24 clips; please report confidence intervals, inter-rater agreement, or statistical significance tests, since the observed differences (e.g., 4.12 vs. 3.98) may not be significant.
  5. [Section 4.5, Fig. 4] The gate visualization shows a single held-out utterance; please state how representative this trajectory is and, ideally, report mean gate statistics aggregated over the whole test set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the phoneme-guidance claim rests on held-out LMD comparisons and ablations, and the only self-citations are peripheral related-work references.

full rationale

PD-GS's central claim is that injecting time-aligned phoneme tokens through the gated Linguistic Fusion Module improves lip articulation. The derivation chain is empirical rather than definitional: phoneme embeddings and the gate are learned under the image reconstruction and landmark consistency losses (Eqs. 3-5), and the claim is tested on held-out HDTF and VoxCeleb2 clips against external baselines and ablations (Tables 1, 2, and 4). No parameter is fitted to the reported LMD values; the reported LMD is an evaluation metric computed on test data. The overlap between the landmark consistency loss (Eq. 4) and the LMD metric is an evaluation-alignment concern, not a circular derivation: the audio-only ablation is trained under the same landmark loss, so the reported LMD difference (2.73 vs. 2.66) is not forced by construction and still provides evidence about the contribution of the phoneme pathway. Self-citations [10], [22], and [23] appear only in related-work discussion and are not load-bearing for the main result; no uniqueness theorem or ansatz is imported from the authors' prior work. The ASR+MFA phoneme pipeline is external and is acknowledged as offline preprocessing. The absence of error bars is a robustness/correctness concern, not circularity. Overall, the paper's derivation is self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or ontological entities; the LFM is a network module. The central claim depends on the accuracy of ASR/MFA alignments, the reliability of landmark supervision, and several hand-chosen hyperparameters that are not accompanied by sensitivity analysis.

free parameters (3)
  • Phoneme vocabulary size V=40 and embedding dimension 64 = V=40, dim=64
    Hand-chosen in Section 3.5; no sensitivity analysis is reported.
  • Loss weighting coefficients lambda_L1, lambda_LPIPS, lambda_landmark = not reported in main text
    Balance image reconstruction, perceptual, and landmark terms (Eq. 3-5); chosen by hand and not documented.
  • Training iteration counts (30k Stage 1, 100k Stage 2) = 30k / 100k
    Reported in Section 4.1, but no validation criterion or sensitivity study is given.
assumptions (4)
  • domain assumption ASR+forced-alignment phoneme labels are correct and frame-aligned for any driving audio.
    Required for the LFM to have a trustworthy phonetic signal; errors would misguide the gate (Section 3.5, 'Phoneme extraction').
  • domain assumption HuBERT features provide sufficient continuous prosodic and audio context for expression and head motion.
    The expression stream relies entirely on HuBERT features (Section 3.5, 'Unified audio representation').
  • domain assumption The pre-trained 2D landmark detector provides reliable ground truth for lip geometry.
    Landmark consistency loss (Eq. 4) and the LMD metric both depend on detector accuracy.
  • standard math 3DMM fitting and 3DGS adaptive densification provide a valid canonical face representation.
    Stage 1 construction follows standard practices in AD-NeRF, GaussianTalker and GeneFace (Section 3.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of PD-GS: Phoneme-Driven 3DGS for Audio-Driven Talking Heads." pith.science (2026). https://pith.science/paper/2M232OPO

@misc{pith2026260805218,
  author       = {Pith},
  title        = {Pith review of: PD-GS: Phoneme-Driven 3DGS for Audio-Driven Talking Heads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2M232OPO}},
  note         = {Machine review of arXiv:2608.05218}
}
read the original abstract

3D Gaussian Splatting (3DGS) enables fast, photorealistic talking-head rendering, yet accurate lip articulation remains elusive: mouth motion is often over-smoothed and may violate hard articulatory constraints such as bilabial closures, producing the notorious ``leaky mouth'' artifact. A key difficulty is that brief, discrete articulatory events are inferred from a continuous acoustic embedding under a regression objective, which biases predictions toward averaged mouth configurations. While modern self-supervised speech encoders provide rich prosodic and phonetic cues, they do not provide an explicit, frame-aligned linguistic target that reliably disambiguates closure-level events. We propose \textbf{Phoneme-Driven Gaussian Splatting (PD-GS)}, which augments a 3DGS talker with time-aligned phoneme tokens obtained from an automatic ASR and forced-alignment pipeline. Our core component, the \textbf{Linguistic Fusion Module (LFM)}, adaptively fuses continuous audio context with discrete phoneme embeddings through a learned gate, allowing the model to preserve smooth audio-driven dynamics while strengthening phoneme guidance on articulation-critical segments. PD-GS is trained purely from monocular video using image reconstruction and lip landmark supervision. On HDTF, PD-GS achieves the best lip geometry among the compared baselines (LMD 2.66) and qualitatively reduces closure violations in challenging phoneme sequences, yielding more linguistically faithful neural avatars.

Figures

Figures reproduced from arXiv: 2608.05218 by the authors.

Figure 1
Figure 1. Dual-Stream Motion Generator. Two complementary streams drive a 3D Gaussian avatar: a prosody stream predicts expression blendshape weights from HuBERT audio features, and an articulation stream fuses time-aligned phonemes through a Linguistic Fusion Module (LFM) with a learned gating network to produce precise lip motions, enabling photorealistic rendering with linguistically faithful articulation. 3D Gaussian Spla… view at source ↗
Figure 2
Figure 2. Overview of our Phoneme-Driven Gaussian Splatting (PD-GS) framework. Stage 1: Canonical avatar construction. From the training video Igt, we fit a 3DMM to recover camera parameters and a neutral facial shape, initializing the canonical 3D Gaussian avatar 𝐺canon. Stage 2: Dual-stream motion learning. A unified HuBERT encoder extracts audio features Faudio from A, shared by an expression stream (blendshape deformation… view at source ↗
Figure 3
Figure 3. Qualitative analysis of articulation errors on challenging phonemes. We highlight common failure modes of audio￾driven baselines: closure failure (incomplete lip sealing for bilabial stops, e.g., [p], producing “leaky mouth”), phonetic ambiguity (distinct vowels, e.g., [æ] and [3], rendered with similarly open mouths), and weak rounding (insufficient lip protrusion, e.g., [ju:]). In contrast, ours produces clearer c… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Mean gate activation over time on a held-out utterance. The gate 𝑡 ̄𝑔 = 1 𝐷 ∑𝑗 𝑔𝑡,𝑗 varies systematically across aligned phoneme intervals instead of collapsing to a constant weight, showing that the LFM adaptively modulates phoneme cues, with local increases near arti…
Figure 5
Figure 5. Figure 5: Phoneme consistency across identities. For the same phoneme (columns), PD-GS generates consistent and physically correct articulations across three subjects (rows), while naturally adapting to each individual’s facial geometry [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 32 canonical work pages

  1. [1]

    Shivangi Aneja, Artem Sevastopolsky, Tobias Kirschstein, Justus Thies, Angela Dai, and Matthias Nießner. 2024. GaussianSpeech: Audio-Driven Gaussian Avatars.arXiv preprint arXiv:2411.18675(2024)

    arXiv 2024

  2. [2]

    Blanz and T

    V. Blanz and T. Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. InProceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH)

    work page 1999

  3. [3]

    Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004. Analysis of emotion recognition using facial expressions, speech and multimodal information. InProceedings of the 6th interna- tional conference on Multimodal interfaces. 205–211

    work page 2004

  4. [4]

    Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013. Facewarehouse: A 3d facial expression database for visual computing.IEEE Transactions on Visualization and Computer Graphics20, 3 (2013), 413–425

    work page 2013

  5. [5]

    Bo Chen, Shoukang Hu, Qi Chen, Chenpeng Du, Ran Yi, Yanmin Qian, and Xie Chen. 2024. GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splatting.arXiv preprint arXiv:2404.19040(2024)

    arXiv 2024

  6. [6]

    K. Cho, J. Lee, H. Yoon, Y. Hong, J. Ko, S. Ahn, and S. Kim. 2024. GaussianTalker: Real-time talking head synthesis with 3D Gaussian Splatting. In Proceedings of the 32nd ACM International Conference on Multimedia

    work page 2024

  7. [7]

    Cudeiro, T

    D. Cudeiro, T. Bolkart, C. Laidlaw, A. Ranjan, and M. J. Black. 2019. Capture, Learning, and Synthesis of 3D Speaking Styles. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2019

  8. [8]

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. 2025. Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 21086–21095

    work page 2025

Show all 38 references
  1. [9]

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. 2022. FaceFormer: Speech-Driven 3D Facial Animation with Transform- ers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  2. [10]

    Ao Fu, Ziqi Ni, and Yi Zhou. 2025. Dual Audio-Centric Modality Coupling for Talking Head Generation. In2025 International Conference on Virtual Reality and Visualization (ICVRV). 884–889.doi:10.1109/ICVRV67992.2025.00155

    2025

  3. [11]

    Peng Gao, Sicheng Song, Chen Li, Yang Liu, Zichao Liu, and Ying Xu. 2024. MimicTalk: Generalizable Talking Head Synthesis in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2024

  4. [12]

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. 2021. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. InProceedings of the IEEE/CVF international conference on computer vision. 5784–5794

    2021

  5. [13]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self- supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processin...

    2021

  6. [14]

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3D Gaussian splatting for real-time radiance field rendering. ACM Trans. Graph.42, 4 (2023), 139–1

    2023

  7. [15]

    R Leanderson, A Persson, and S Öhman. 1971. Electromyographic studies of facial muscle activity in speech.Acta oto-laryngologica72, 1-6 (1971), 361–369

    1971

  8. [16]

    Jiye Lee, Chenghui Li, Linh Tran, Shih-En Wei, Jason Saragih, Alexander Richard, Hanbyul Joo, and Shaojie Bai. 2025. Audio Driven Real-Time Facial Animation for Social Telepresence.arXiv preprint arXiv:2510.01176(2025)

    2025

  9. [17]

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. 2024. Talkinggaussian: Structure-persistent 3d talking head synthesis via gaussian splatting. InEuropean Conference on Computer Vision. Springer, 127–145

    2024

  10. [18]

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. 2023. Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis. InProceedings of the IEEE/CVF International Conference on Computer Vision

    2023

  11. [19]

    Tianqi Li, Ruobing Zheng, Minghui Yang, Jingdong Chen, and Ming Yang. 2024. Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis.arXiv preprint arXiv:2411.19509(2024)

    2024

  12. [20]

    Montrealforcedaligner:Trainabletext-speech alignment using kaldi

    MichaelMcAuliffe,MichaelaSocolof,SarahMihuc,MichaelWagner,andMorganSonderegger.2017. Montrealforcedaligner:Trainabletext-speech alignment using kaldi.. InInterspeech, Vol. 2017. 498–502

    2017

  13. [21]

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis.Commun. ACM65, 1 (2021), 99–106

    2021

  14. [22]

    FREAK:Frequency-modulatedHigh-fidelityandReal-timeAudio-drivenTalkingPortraitSynthesis.In Proceedings of the 2025 International Conference on Multimedia Retrieval (ICMR ’25)

    ZiqiNi,AoFu,andYiZhou.2025. FREAK:Frequency-modulatedHigh-fidelityandReal-timeAudio-drivenTalkingPortraitSynthesis.In Proceedings of the 2025 International Conference on Multimedia Retrieval (ICMR ’25). Association for Computing Machinery, New York, NY, USA, 1036–1044. PD-GS: ...

    2025

  15. [23]

    Ziqi Ni, Yuanzhi Liang, Rui Li, Yi Zhou, Haibin Huang, Chi Zhang, and Xuelong Li. 2026. Seeing What Matters: Visual Preference Policy Optimiza- tion for Visual Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 27260–27269

    2026

  16. [24]

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. 2024. SyncTalk: The Devil Is in the Synchronization for Talking Head Synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2024

  17. [25]

    Pascal Perrier, Rafael Laboissière, Christian Abry, and Shinji Maeda. 1997. Speech production: Models and data.Speech Communication22, 2 (1997), 89–92. doi:10.1016/S0167-6393(97)00028-9

    1997 doi

  18. [26]

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020. A lip sync expert is all you need for speech to lip generation in the wild. InProceedings of the 28th ACM international conference on multimedia. 484–492

    2020

  19. [27]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning. PMLR, 28492–28518

    2023

  20. [28]

    J. Tang, K. Wang, H. Zhou, X. Chen, D. He, T. Hu, J. Liu, G. Zeng, and J. Wang. 2022. Real-Time Neural Radiance Talking Portrait Synthesis via Audio-Spatial Decomposition. InEuropean Conference on Computer Vision

    2022

  21. [29]

    Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliakbarian, Darren Cosker, Christian Theobalt, and Justus Thies. 2023. Imitator: Personalized Speech-driven 3D Facial Animation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    2023

  22. [30]

    Jie Wang, Jiu-Cheng Xie, Xianyan Li, Feng Xu, Chi-Man Pun, and Hao Gao. 2023. GaussianHead: High-fidelity Head Avatars with Learnable Gaussian Derivation.arXiv preprint arXiv:2312.01632(2023)

    2023 arXiv

  23. [31]

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. 2023. CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  24. [32]

    MuseTalk:AFull-BodyVideo-DrivenTalkingHeadFramework with Expressive Speech Styles

    ZhitongYang,ZhongWang,ShiguangLiu,ButianYu,XinYan,andHaoShao.2024. MuseTalk:AFull-BodyVideo-DrivenTalkingHeadFramework with Expressive Speech Styles. arXiv preprint arXiv:2401.06820

    2024 arXiv

  25. [33]

    Z. Ye, Z. Jiang, Y. Ren, J. Liu, J. He, and Z. Zhao. 2023. GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2023

  26. [34]

    Theunreasonableeffectivenessof deepfeaturesasaperceptual metric

    RichardZhang,PhillipIsola,AlexeiAEfros,EliShechtman, andOliverWang.2018. Theunreasonableeffectivenessof deepfeaturesasaperceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition. 586–595

    2018

  27. [35]

    Sadtalker:Learningrealistic3dmotion coefficients for stylized audio-driven single image talking face animation

    WenxuanZhang,XiaodongCun,XuanWang,YongZhang,XiShen,YuGuo,YingShan,andFeiWang.2023. Sadtalker:Learningrealistic3dmotion coefficients for stylized audio-driven single image talking face animation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2023

  28. [36]

    Zhang, Z

    Z. Zhang, Z. Hu, W. Deng, C. Fan, T. Lv, and Y. Ding. 2023. DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video. InProceedings of the AAAI Conference on Artificial Intelligence

    2023

  29. [37]

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. 2021. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3661–3670

    2021

  30. [38]

    Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. 2023. Identity-preserving Talking Face Gener- ation with Landmark and Appearance Priors. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.