Pith. sign in

REVIEW 5 major objections 6 minor 52 references

PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PointTalk claims that adding a speech-generated dynamic lip point cloud to 3D Gaussian deformation improves both image quality and lip sync in talking-head synthesis.

desk verdict A plausible 3DGS talking-head method with a genuinely new lip-point-cloud conditioning trick, but the abstract's lip-sync claim overreaches and the evaluation lacks the transparency to back it. read the letter →

arxiv 2412.08504 v1 pith:MSEDHGVX submitted 2024-12-11 cs.SD cs.AIcs.GRcs.MMeess.AS

Yifan Xie , Tao Feng , Xin Zhang , Xiangyang Luo , Zixuan Guo , Weijiang Yu , Heng Chang , Fei Ma
show 1 more author
Fei Richard Yu
This is my paper · ORCID
classification cs.SDcs.AIcs.GRcs.MMeess.AS
keywords talkingheadsynthesis3DGaussiansplattingaudio-drivenanimationlippointcloudsynchronizationcross-modalcontrastivelearningdynamicdifferenceencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PointTalk is a talking-head synthesis method built on 3D Gaussian Splatting. The paper's central claim is that deforming a static Gaussian head with both speech audio and a dynamic lip point cloud generated from that audio yields higher image fidelity and more accurate audio-lip synchronization than prior NeRF- and Gaussian-based methods. In the paper's experiments, PointTalk reports the best PSNR (32.770), LPIPS (0.0337), LMD (2.818), LSE-D (7.383), and LSE-C (7.165) among the compared methods on head reconstruction, and the best LSE-C on two out-of-distribution audio clips. Ablations that remove the dynamic difference encoder, the contrastive loss, or the attention module all degrade lip-sync metrics, which supports the paper's claim that the lip point cloud and its alignment with audio are doing real work. If the claim is right, audio-generated point clouds are a viable conditioning signal for real-time Gaussian avatars.

What carries the argument

The load-bearing object is the dynamic lip point cloud and the feature pipeline around it. Audio goes into the Audio2Point module, which produces frames of lip points; a multi-resolution hash grid encodes each frame, and the dynamic difference encoder forms $F_p$ by concatenating neighboring-frame differences $E_{\mathrm{point}}(P_{t+1}) - E_{\mathrm{point}}(P_t)$. The audio-point enhancement module applies cross-modal contrastive learning, with loss $L_{CL}$ in Eq. (6), plus external attention, so the audio and point features are aligned and correlated. These enhanced features feed an AdaIN-style adaptive MLP that predicts deformation parameters $\Delta x, \Delta r, \Delta s$ for each Gaussian primitive, which the 3D Gaussian rasterizer renders. The mechanism that carries the argument is the coupling of a topological, motion-sensitive point-cloud representation with audio features before deformation.

What would settle it

Drive the same static Gaussian head with identical audio features but replace the dynamic lip point cloud with a time-shifted version of itself or with points from a different speaker's lips; if LSE-C and LMD stay at the same level, the reported synchronization gain comes from the audio branch or the mesh regressor rather than from the lip point cloud.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an audio-driven dynamic lip point cloud can serve as a strong conditional signal for 3D Gaussian talking heads. The method first builds a static field of Gaussian primitives with a tri-plane spatial feature, then predicts per-primitive deformation from audio features and point-cloud features. The Audio2Point module converts speech into a sequence of lip point clouds via a mesh regressor; a multi-resolution hash grid encodes each frame's topological structure, and a dynamic difference encoder concatenates inter-frame feature differences so the deformation network sees motion, not just shape. An audio-point enhancement module synchronizes the two modalities through a cross-modal contrastive loss and mixes them with the spatial features by Hadamard products, and an AdaIN-style residual block converts the fused features into scale and shift parameters that deform the Gaussians. The paper argues that this composition, rather than any single component, is what produces the reported image-quality and lip-sync gains.

Load-bearing premise

The argument depends on the Audio2Point module producing a lip point cloud that is geometrically accurate and adds information beyond what the audio branch already provides; the module's details are deferred to the supplementary material, so the main text cannot establish this.

Editorial extensions

If this is right

  • If PointTalk is right, a mesh-guided lip point cloud generated from audio can be added to existing 3D Gaussian deformation pipelines without slowing training much: the paper reports about one hour of training and 85 FPS inference.
  • Lip-sync metrics improve on out-of-distribution audio, suggesting the audio-point alignment generalizes beyond the training speaker's voice.
  • The dynamic difference encoder's inter-frame differencing shows that motion of the lip point cloud, not just its per-frame geometry, helps the deformation network.
  • The ablation results indicate that the contrastive synchronization loss and the cross-modal attention are each necessary for the reported sync quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: replace the Audio2Point-generated cloud with ground-truth or time-shifted lip points while keeping audio features identical; if sync metrics do not change, the point cloud's contribution is largely regularization rather than new phonetic information.
  • The method inherits the mesh regressor's lip-motion prior, so its generalization may be bounded by that regressor; evaluating with different regressors could separate the point-cloud contribution from the prior.
  • The same audio-point contrastive design could be applied to other face regions, such as eyes or brows, or to body and gesture animation, where a generated point cloud from audio may similarly disambiguate motion.
  • Because the contrastive loss aligns audio with a cloud produced from that same audio, the loss may partly be enforcing self-consistency of the Audio2Point transform; ablating with a frozen, separately trained lip regressor would clarify this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces PointTalk, a 3D Gaussian Splatting framework for audio-driven talking head synthesis. The method constructs a static Gaussian head, extracts audio features with an ASR encoder, and generates a dynamic lip point cloud from the same audio via an Audio2Point module. A dynamic difference encoder and an audio-point enhancement module (combining cross-modal contrastive learning and external attention) produce deformation parameters, which are applied through an AdaIN-style adaptive MLP to deform the Gaussians. Experiments on a collected dataset compare against 2D and 3D baselines, reporting reconstruction metrics, lip-sync metrics, FPS, a user study, and ablations. The central claim is that PointTalk achieves superior high-fidelity and audio-lip synchronization relative to previous methods.

Significance. If the empirical claims are confirmed, PointTalk is a coherent engineering extension of 3DGS talking-head methods: it injects a geometric lip condition into the deformation pipeline, runs in real time, and reports strong visual quality relative to 3D baselines. The ablation study covers the main design choices (encoder type, hash grid hyperparameters, dynamic difference encoder, contrastive loss, attention), and the user study is a useful complement. However, the evidence in the paper does not yet support the unqualified superiority claim: the lip-sync metrics in Tables 1 and 2 are worse than Wav2Lip's on the reported criteria, the full-model ablation row disagrees with Table 1, and the dataset and evaluation protocol are underspecified. These issues are correctable in revision.

major comments (5)
  1. [Abstract; Tables 1 and 2] The abstract's claim of 'superior ... audio-lip synchronization ... compared to previous methods' is contradicted by the paper's own lip-sync metrics. In Table 2, Wav2Lip achieves LSE-D/LSE-C of 7.896/7.393 on Audio A and 6.760/9.259 on Audio B, while PointTalk achieves 8.406/6.427 and 8.331/7.018; since lower LSE-D and higher LSE-C are better, Wav2Lip is better on LSE-C in both clips and better on LSE-D in Audio B. Table 1 shows the same pattern (Wav2Lip: 7.326/8.363 vs PointTalk: 7.383/7.165). The abstract and conclusion should be explicitly scoped to 3D-based methods, or the central claim as written is not supported by the reported evidence.
  2. [Table 3 vs Table 1] The 'PointTalk' row of Table 3 (PSNR 32.704, LPIPS 0.0365, FID 7.194, LMD 2.773, LSE-D 7.367, LSE-C 7.079) does not match the PointTalk row of Table 1 (32.770, 0.0337, 7.331, 2.818, 7.383, 7.165), although both are described as the full model under the head-reconstruction setting. The authors must specify the exact evaluation protocol and explain the discrepancy; without this, the internal consistency of the quantitative evaluation is in question.
  3. [Experimental Settings] The dataset description gives average clip length and resolution but omits the number of subjects, the number of clips, the train/test split, and the exact composition relative to the source datasets (RAD-NeRF, GeneFace, ER-NeRF). All metrics are single runs without error bars or significance tests, so small improvements such as PSNR 32.704 vs TalkingGaussian's 32.398 cannot be assessed. The LSE-D/LSE-C protocol also does not state whether mouth crops are used or which pretrained sync network is used, which matters for the comparison with 2D methods.
  4. [Eqs. (4)-(6)] Eq. (4) defines Fp in R^{(T-1)xF} from differences of neighboring frame encodings, while Eqs. (5)-(6) use per-frame features F'_p in R^{T x F} indexed by t=1,...,T. The paper does not specify whether the contrastive loss uses the dynamic-difference features or the per-frame features, nor how Fp and F'_p are combined in the enhancement module. This ambiguity affects the reproducibility of the central training objective.
  5. [Audio-Point Enhancement; Eqs. (5)-(6)] Because the lip point cloud is generated from the same audio signal Fa through Audio2Point, the contrastive loss in Eqs. (5)-(6) aligns audio features with a deterministic function of themselves. The paper should clarify what non-trivial cross-modal correspondence is being learned and whether the improvement from CCL in Table 3 (LSE-D 7.786 to 7.367, LSE-C 6.455 to 7.079) comes from genuine synchronization or from an auxiliary training signal. This does not invalidate the method, but it weakens the claim that the module synchronizes two independent modalities.
minor comments (6)
  1. [Audio2Point module; Figure 2] The Audio2Point module is central to the contribution, but its architecture, training loss, and the 'Lip Selection' step in Figure 2 are only described by reference to the supplementary material; please provide a self-contained description in the main text.
  2. [Figure 5] The user-study figure does not report confidence intervals or the number of ratings per clip, and the claim of a 'over 20%' margin would be strengthened by a statistical test.
  3. [Loss Function; Eq. (10)] Equation (10) contains a typo: 'construsted' should be 'constructed'.
  4. [Method; Eq. (2)] The default values of the hash grid levels L and feature dimension F are first revealed in the ablation study (L=8, F=4); state them where Eq. (2) is introduced.
  5. [Table 1] The Time column mixes training time and inference FPS in one column; separate them or label the units clearly.
  6. [Figure 4] The qualitative comparison includes Wav2Lip and DINet but not TalkLip, even though TalkLip appears in the quantitative tables; either add it or note why it was omitted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the audio-point contrastive loss is an internal training objective, and the paper's lip-sync and quality claims are assessed against external baselines and metrics.

full rationale

PointTalk is an empirical system paper: the head reconstruction and lip-sync claims are supported by quantitative comparisons (PSNR, LPIPS, FID, LMD, LSE-D, LSE-C) against eight baselines on held-out test frames and out-of-distribution audio clips. The only candidate for built-in circularity is the Audio-Point Enhancement module's contrastive loss in Eqs. (5)-(6), which aligns audio features with point-cloud features of the same frame. Since the dynamic lip point cloud P is generated from the audio by the Audio2Point module, this loss enforces internal cross-modal consistency between two views of the same input rather than measuring a prediction against independent ground truth. It does not by construction produce the reported LSE-D/LSE-C numbers, which are computed with the standard Wav2Lip lip-sync expert on held-out audio and are not forced by the loss. The paper cites the authors' own earlier point-cloud works (Xie et al. 2023, 2024), but only in a survey sentence about point cloud learning; no load-bearing claim, uniqueness theorem, or ansatz is imported from those self-citations. The abstract's unqualified 'superior' claim is in tension with Table 2, where Wav2Lip achieves higher LSE-C on both OOD clips, but this is an internal evaluation/overclaim issue, not a circularity of derivation, and is excluded from this score under the provided rules.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on several domain assumptions about the fidelity of the audio-to-mesh prior, the validity of lip-sync metrics, and the sufficiency of deformation-only Gaussian animation. The free hyperparameters are not fully reported, and the key assumption that the audio-generated lip point cloud adds information beyond the audio itself is not independently tested.

free parameters (4)
  • contrastive temperature tau = not reported
    Temperature in Eq. (5) controls the sharpness of the audio-point contrastive loss, but its value is not given.
  • loss weights lambda1, lambda2, lambda3 = not reported
    Weights in Eq. (10) balance the D-SSIM, LPIPS, and contrastive terms; values are omitted from the main text.
  • hash grid levels L and feature dimension F = L=8, F=4
    Selected via ablation in Table 3 as the best of four settings; these are hand-chosen hyperparameters that affect all results.
  • number of Gaussian primitives and training schedule = not reported
    3D Gaussian initialization and deformation training hyperparameters are not stated.
assumptions (4)
  • domain assumption The Audio2Point mesh regressor produces lip point clouds whose geometry is a reliable proxy for true lip motion.
    The entire lip point cloud condition depends on this; module details are only in the supplementary material.
  • domain assumption LSE-D and LSE-C, computed with a pretrained lip-sync expert, are valid measures of audio-lip synchronization for this task.
    The paper uses these metrics as primary evidence of lip-sync improvement without validating them on the test distribution.
  • domain assumption A static 3D Gaussian field plus per-frame affine deformation can represent all needed talking head motion.
    The method does not model color or opacity changes during speech, assuming deformation alone suffices.
  • domain assumption Multi-resolution hash encoding captures the topological structure of a point cloud with a few hundred points.
    The paper argues hash collisions are rare because the point count is small, but does not provide a theoretical guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis." pith.science (2026). https://pith.science/paper/MSEDHGVX

@misc{pith2026241208504,
  author       = {Pith},
  title        = {Pith review of: PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MSEDHGVX}},
  note         = {Machine review of arXiv:2412.08504}
}
read the original abstract

Talking head synthesis with arbitrary speech audio is a crucial challenge in the field of digital humans. Recently, methods based on radiance fields have received increasing attention due to their ability to synthesize high-fidelity and identity-consistent talking heads from just a few minutes of training video. However, due to the limited scale of the training data, these methods often exhibit poor performance in audio-lip synchronization and visual quality. In this paper, we propose a novel 3D Gaussian-based method called PointTalk, which constructs a static 3D Gaussian field of the head and deforms it in sync with the audio. It also incorporates an audio-driven dynamic lip point cloud as a critical component of the conditional information, thereby facilitating the effective synthesis of talking heads. Specifically, the initial step involves generating the corresponding lip point cloud from the audio signal and capturing its topological structure. The design of the dynamic difference encoder aims to capture the subtle nuances inherent in dynamic lip movements more effectively. Furthermore, we integrate the audio-point enhancement module, which not only ensures the synchronization of the audio signal with the corresponding lip point cloud within the feature space, but also facilitates a deeper understanding of the interrelations among cross-modal conditional features. Extensive experiments demonstrate that our method achieves superior high-fidelity and audio-lip synchronization in talking head synthesis compared to previous methods.

Figures

Figures reproduced from arXiv: 2412.08504 by the authors.

Figure 1
Figure 1. Overview of PointTalk. Utilizing the static Gaussian field to optimize the coarse Gaussian head from a random point [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of the Audio2Point module. track, where T denotes the frame count and F represents the feature dimension. For additional details, please refer to the supplementary materials. Lip Point Cloud Branch. Inspired by mesh-based meth￾ods (Fan et al. 2022; Peng et al. 2023), we design an Au￾dio2Point module to capture an additional dynamic lip point cloud. The pipeline of the Audio2Point module is depicted in … view at source ↗
Figure 3
Figure 3. The detailed structure of the Audio-Point Enhance [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of talking head synthesis by different methods. PointTalk has the best visual effect on lip [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Visualization of the ablation study. Setting PSNR↑ LPIPS↓ FID↓ LMD↓ LSE-D↓ LSE-C↑ GNN 31.011 0.0416 9.353 3.655 8.135 6.253 Tri-Plane 32.390 0.0389 7.319 2.873 7.636 6.816 Hash Grid 32.704 0.0365 7.194 2.773 7.367 7.079 L=32 F=1 32.515 0.0378 7.324 2.915 7.564 6.818 L=…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 30 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Amodei, D.; Ananthanarayanan, S.; Anubhai, R.; Bai, J.; Battenberg, E.; Case, C.; Casper, J.; Catanzaro, B.; Cheng, Q.; Chen, G.; et al. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In International Conference on Machine Learning, 173--182. PMLR

    work page 2016

  4. [4]

    Blanz, V.; and Vetter, T. 2023. A morphable model for the synthesis of 3D faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, 157--164

    work page 2023

  5. [5]

    R.; Lin, C

    Chan, E. R.; Lin, C. Z.; Chan, M. A.; Nagano, K.; Pan, B.; De Mello, S.; Gallo, O.; Guibas, L. J.; Tremblay, J.; Khamis, S.; et al. 2022. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16123--16133

    work page 2022

  6. [6]

    Chen, G.; Wang, M.; Yang, Y.; Yu, K.; Yuan, L.; and Yue, Y. 2024. Pointgpt: Auto-regressively generative pre-training from point clouds. Advances in Neural Information Processing Systems, 36

    work page 2024

  7. [7]

    S.; and Zisserman, A

    Chung, J. S.; and Zisserman, A. 2017. Out of time: automated lip sync in the wild. In Computer Vision--ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, 251--263. Springer

    2017

  8. [8]

    Fan, Y.; Lin, Z.; Saito, J.; Wang, W.; and Komura, T. 2022. Faceformer: Speech-driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18770--18780

    2022

Show all 52 references
  1. [9]

    Guo, M.-H.; Liu, Z.-N.; Mu, T.-J.; and Hu, S.-M. 2022. Beyond self-attention: External attention using two linear layers for visual tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5): 5436--5447

    2022

  2. [10]

    Guo, Y.; Chen, K.; Liang, S.; Liu, Y.-J.; Bao, H.; and Zhang, J. 2021. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 5784--5794

    2021

  3. [11]

    Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30

    2017

  4. [12]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33: 6840--6851

    2020

  5. [13]

    H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A

    Hsu, W.-N.; Bolte, B.; Tsai, Y.-H. H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 3451--3460

    2021

  6. [14]

    Huang, X.; and Belongie, S. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, 1501--1510

    2017

  7. [15]

    Ji, X.; Zhou, H.; Wang, K.; Wu, Q.; Wu, W.; Xu, F.; and Cao, X. 2022. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings, 1--10

    2022

  8. [16]

    C.; Cao, X.; and Xu, F

    Ji, X.; Zhou, H.; Wang, K.; Wu, W.; Loy, C. C.; Cao, X.; and Xu, F. 2021. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14080--14089

    2021

  9. [17]

    Kenton, J. D. M.-W. C.; and Toutanova, L. K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, 4171--4186

    2019

  10. [18]

    Kerbl, B.; Kopanas, G.; Leimk \"u hler, T.; and Drettakis, G. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph., 42(4): 139--1

    2023

  11. [19]

    Kim, H.; Garrido, P.; Tewari, A.; Xu, W.; Thies, J.; Niessner, M.; P \'e rez, P.; Richardt, C.; Zollh \"o fer, M.; and Theobalt, C. 2018. Deep video portraits. ACM transactions on graphics (TOG), 37(4): 1--14

    2018

  12. [20]

    Xie, Y.; Zhu, J.; Li, S.; and Shi, P. 2023. Cross-modal information-guided network using contrastive learning for point cloud registration. IEEE Robotics and Automation Letters, 9(1): 103--110

    2023

  13. [21]

    Li, J.; Zhang, J.; Bai, X.; Zheng, J.; Ning, X.; Zhou, J.; and Gu, L. 2024. TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting. arXiv preprint arXiv:2404.15264

    2024 arXiv

  14. [22]

    Li, J.; Zhang, J.; Bai, X.; Zhou, J.; and Gu, L. 2023. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 7568--7578

    2023

  15. [23]

    Ma, Y.; Zhang, S.; Wang, J.; Wang, X.; Zhang, Y.; and Deng, Z. 2023. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767

    2023 arXiv

  16. [24]

    P.; Tancik, M.; Barron, J

    Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; and Ng, R. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99--106

    2021

  17. [25]

    M \"u ller, T.; Evans, A.; Schied, C.; and Keller, A. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4): 1--15

    2022

  18. [26]

    T.; Bouaziz, S.; Goldman, D

    Park, K.; Sinha, U.; Hedman, P.; Barron, J. T.; Bouaziz, S.; Goldman, D. B.; Martin-Brualla, R.; and Seitz, S. M. 2021. HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG), 40(6): 1--12

    2021

  19. [27]

    Peng, Z.; Hu, W.; Shi, Y.; Zhu, X.; Zhang, X.; He, J.; Liu, H.; and Fan, Z. 2024. SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  20. [28]

    Peng, Z.; Luo, Y.; Shi, Y.; Xu, H.; Zhu, X.; Liu, H.; He, J.; and Fan, Z. 2023. Selftalk: A self-supervised commutative training diagram to comprehend 3d talking faces. In Proceedings of the 31st ACM International Conference on Multimedia, 5292--5301

    2023

  21. [29]

    P.; and Jawahar, C

    Prajwal, K.; Mukhopadhyay, R.; Namboodiri, V. P.; and Jawahar, C. 2020. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, 484--492

    2020

  22. [30]

    R.; Su, H.; Mo, K.; and Guibas, L

    Qi, C. R.; Su, H.; Mo, K.; and Guibas, L. J. 2017 a . Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 652--660

    2017

  23. [31]

    R.; Yi, L.; Su, H.; and Guibas, L

    Qi, C. R.; Yi, L.; Su, H.; and Guibas, L. J. 2017 b . Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 30

    2017

  24. [32]

    Qi, Z.; Dong, R.; Fan, G.; Ge, Z.; Zhang, X.; Ma, K.; and Yi, L. 2023. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, 28223--28243. PMLR

    2023

  25. [33]

    Qian, S.; Kirschstein, T.; Schoneveld, L.; Davoli, D.; Giebenhain, S.; and Nie ner, M. 2024. Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20299--20309

    2024

  26. [34]

    Shen, S.; Li, W.; Huang, X.; Zhu, Z.; Zhou, J.; and Lu, J. 2023 a . SD-NeRF: Towards Lifelike Talking Head Animation via Spatially-adaptive Dual-driven NeRFs. IEEE Transactions on Multimedia

    2023

  27. [35]

    Shen, S.; Zhao, W.; Meng, Z.; Li, W.; Zhu, Z.; Zhou, J.; and Lu, J. 2023 b . Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1982--1991

    2023

  28. [36]

    M.; and Kemelmacher-Shlizerman, I

    Suwajanakorn, S.; Seitz, S. M.; and Kemelmacher-Shlizerman, I. 2017. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (TOG), 36(4): 1--13

    2017

  29. [37]

    Tang, J.; Wang, K.; Zhou, H.; Chen, X.; He, D.; Hu, T.; Liu, J.; Zeng, G.; and Wang, J. 2022. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368

    2022 arXiv

  30. [38]

    Thies, J.; Elgharib, M.; Tewari, A.; Theobalt, C.; and Nie ner, M. 2020. Neural voice puppetry: Audio-driven facial reenactment. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XVI 16, 716--731. Springer

    2020

  31. [39]

    Wang, J.; Li, X.; Xie, J.; Xu, F.; and Gao, H. 2023 a . GaussianHead: Impressive 3D Gaussian-based Head Avatars with Dynamic Hybrid Neural Field. arXiv e-prints, arXiv--2312

    2023

  32. [40]

    T.; and Li, H

    Wang, J.; Qian, X.; Zhang, M.; Tan, R. T.; and Li, H. 2023 b . Seeing what you said: Talking face generation guided by a lip reading expert. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14653--14662

    2023

  33. [41]

    E.; Bronstein, M

    Wang, Y.; Sun, Y.; Liu, Z.; Sarma, S. E.; Bronstein, M. M.; and Solomon, J. M. 2019. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG), 38(5): 1--12

    2019

  34. [42]

    Xie, Y.; Zhu, J.; Li, S.; Hu, N.; and Shi, P. 2024. HECPG: Hyperbolic Embedding and Confident Patch-Guided Network for Point Cloud Matching. IEEE Transactions on Geoscience and Remote Sensing

    2024

  35. [43]

    Xing, J.; Xia, M.; Zhang, Y.; Cun, X.; Wang, J.; and Wong, T.-T. 2023. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12780--12790

    2023

  36. [44]

    Ye, Z.; He, J.; Jiang, Z.; Huang, R.; Huang, J.; Liu, J.; Ren, Y.; Yin, X.; Ma, Z.; and Zhao, Z. 2023. Geneface++: Generalized and stable real-time audio-driven 3d talking face generation. arXiv preprint arXiv:2305.00787

    2023 arXiv

  37. [45]

    Ye, Z.; Jiang, Z.; Ren, Y.; Liu, J.; He, J.; and Zhao, Z. 2022. GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis. In The Eleventh International Conference on Learning Representations

    2022

  38. [46]

    Ye, Z.; Zhong, T.; Ren, Y.; Yang, J.; Li, W.; Huang, J.; Jiang, Z.; He, J.; Huang, R.; Liu, J.; et al. 2024. Real3d-portrait: One-shot realistic 3d talking portrait synthesis. arXiv preprint arXiv:2401.08503

    2024 arXiv

  39. [47]

    Yu, X.; Tang, L.; Rao, Y.; Huang, T.; Zhou, J.; and Lu, J. 2022. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19313--19322

    2022

  40. [48]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 586--595

    2018

  41. [49]

    Zhang, W.; Cun, X.; Wang, X.; Zhang, Y.; Shen, X.; Guo, Y.; Shan, Y.; and Wang, F. 2023 a . Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2023

  42. [50]

    Zhang, Z.; Hu, Z.; Deng, W.; Fan, C.; Lv, T.; and Ding, Y. 2023 b . Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 3543--3551

    2023

  43. [51]

    Zhang, Z.; Li, L.; Ding, Y.; and Fan, C. 2021. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3661--3670

    2021

  44. [52]

    Zhong, W.; Fang, C.; Cai, Y.; Wei, P.; Zhao, G.; Lin, L.; and Li, G. 2023. Identity-preserving talking face generation with landmark and appearance priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9729--9738

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.