Pith. sign in

REVIEW 4 major objections 5 minor 86 references

ViDS: Video Diffusion Shader using 3D Face Tracking

T0 review · 4 major / 5 minor · reviewed 2026-07-31 · deepseek-v4-flash

Pith's one-line read A video diffusion model can act as a neural shader for portrait animation when conditioned on dense, pixel-aligned 3D face normal maps rather than sparse landmarks or implicit motion latents.

desk verdict Solid, well-ablated systems paper; the dense 3DMM-normal-map conditioning is a real integration, but the motion metrics don't back the abstract's expression-control claim. read the letter →

arxiv 2607.24124 v1 pith:D4EP55EQ submitted 2026-07-27 cs.CV

classification cs.CV
keywords portraitanimationvideodiffusion3Dmorphablemodelnormalmapconditioningidentitypreservationfacereenactmentautoregressivegenerationclassifier-freeguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ViDS sets out to prove that a video diffusion model can act as a neural shader for portrait animation when it is given dense geometric guidance from a 3D Morphable Model tracker. The core proposal is to render pixel-aligned normal maps from an identity-frozen 3D face mesh animated by a driving video, and concatenate them with the reference image and noisy latents into a pretrained video diffusion transformer. The authors argue this yields finer and more consistent expression and pose control than landmark-based or implicit-latent diffusion baselines, while preserving reference identity and appearance — including non-3DMM regions such as inner mouth and hair. The claim is supported by self- and cross-reenactment benchmarks on VFHQ and Celeb-V-Text, a user study, and ablations that isolate the conditioning signal and tracking quality.

What carries the argument

The central object is the rendered pixel-aligned normal-map sequence from an animated 3DMM mesh. Unlike low-dimensional 3DMM parameters or sparse landmarks, these maps carry dense local surface orientation that is lighting-agnostic and frame-aligned with the reference image; freezing the reference identity shape separates identity from pose and expression. This geometry channel is injected into a pretrained video diffusion transformer via a unified channel-concatenation layer (reference latents, normal-map latents, noisy video latents), and three separate classifier-free guidance branches (identity, geometry, text) are combined into one velocity field. Long sequences are produced by an autor

What would settle it

Run the VFHQ self- and cross-reenactment evaluation while progressively corrupting or replacing the 3DMM tracking (e.g., jittering pose parameters or substituting SHeaP), and check whether identity and geometry metrics degrade as sharply as Table 8 suggests; if a corrupted tracking signal still yields strong CSIM and AKD, the conditioning is not doing the claimed work. A second decisive check is to re-evaluate all methods with a non-3DMM motion-fidelity metric or independent human ratings on expression transfer, since AED/APD share the paper's geometric prior.

Watch

Extended reading notes

Core claim

ViDS claims that accurate monocular 3DMM tracking — specifically the Pixel3DMM tracker — converts expression and pose transfer into a shading problem. An identity-specific mesh is reconstructed from one reference image, then animated with a driving video's pose and expression parameters while shape parameters are frozen to prevent identity leakage; the animated mesh is rendered as normal maps and fed, together with the reference image and a text prompt, into a video diffusion transformer. The study reports that this dense geometric conditioning outperforms sparse-landmark and implicit-latent methods on most metrics in self- and cross-reenactment, with a particular advantage in identity prese

Load-bearing premise

The whole pipeline leans on the assumption that Pixel3DMM tracking is accurate enough that its rendered normal maps faithfully capture the driving video's pose and expression — the paper's own conclusion lists tracking sensitivity as a limitation, and the ablation shows that swapping in SHeaP tracking sharply degrades identity and geometry metrics.

Editorial extensions

If this is right

  • Dense geometric conditioning becomes a workable alternative to landmarks or implicit latents for one-shot portrait animation.
  • Improvements in monocular 3D face tracking quality should translate directly into finer expression and pose transfer, since the normal maps are the sole motion channel.
  • Identity leakage from the driving video is reduced by freezing the reference identity's 3DMM shape parameters, which should help cross-identity and in-the-wild reenactment.
  • The autoregressive overlapping-window scheme extends a pretrained video diffusion model beyond its native temporal window while reducing boundary discontinuities.
  • The method retains photorealistic synthesis of regions the 3DMM does not model, such as inner mouth and hair, because the diffusion prior still generates those details.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation list (tracking sensitivity, slow autoregressive inference, no relighting, limited control beyond 3DMM regions) points to the clearest next steps: more robust tracking and faster sampling would expand the method's practical range.
  • Because the reported motion metrics (AED/APD) are computed with a 3DMM estimator, a re-ranking of methods using a different estimator or purely perceptual motion judgments would test whether the advantage is genuine or partly an artifact of shared geometric priors.
  • A testable extension: swapping the 3DMM tracker at inference for a stronger one should improve the same metrics on the same benchmarks, providing a direct way to measure how much headroom remains in tracking quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ViDS, a portrait animation method that uses Pixel3DMM/FLAME normal maps as dense geometric conditioning for a Wan-based video diffusion model. Identity shape is fixed from the reference image, driving pose/expression animates the mesh, and normal maps are concatenated with reference latents through a unified conditioning layer. Multi-CFG with separate identity/geometry/text guidance and autoregressive overlapping-window inference are used. Experiments on VFHQ and Celeb-V-Text compare to Follow-Your-Emoji, X-Portrait, HunyuanPortrait, and Wan-Animate, with ablations on conditioning signals, architecture, guidance scale, training size, and tracker. The paper reports best or second-best results on most reconstruction and identity metrics, and a user study favors ViDS on all dimensions.

Significance. If the evidence held, ViDS would be a useful and simple design: it injects dense 3DMM geometry into a pretrained video diffusion model with minimal architectural change, and it demonstrates the importance of tracking quality. The ablations are extensive and the training protocol is described in enough detail to reproduce. The main weakness is that the advertised advantage in expression/pose control is not consistently supported by the numerical motion metrics, and the motion metrics themselves share a 3DMM prior with the conditioning signal. The central claim is thus plausible but not conclusively demonstrated.

major comments (4)
  1. [Sec. 4.1, Tables 1 and 2] The abstract claims 'more detailed and consistent expression and pose control', but the only direct expression metric (AED) never favors ViDS. In Table 1 self-reenactment AED is 0.121 for Ours versus 0.113 (HunyuanPortrait), 0.118 (X-Portrait), and 0.118 (Wan-Animate); cross-reenactment AED is 0.298 versus 0.279 (HunyuanPortrait). In Table 2, Ours again has worse AED in both self (0.189 vs 0.161) and cross (0.306 vs 0.298) settings. This should be confronted directly: either temper the claim, show that AED is the wrong yardstick with a validated alternative, or provide a significance test showing the differences are not meaningful.
  2. [Sec. 4, Evaluation Protocol; Sec. 3.1] AED and APD are computed with Deep3DFaceRecon [18], a monocular 3DMM estimator, while the conditioning signal is a rendered 3DMM (Pixel3DMM/FLAME) normal map. This creates a circularity concern: outputs that stay close to the 3DMM manifold may score better on these metrics even if they are not perceptually more faithful. The paper acknowledges the limitation and adds a user study, but the user study is too small (10 videos, 40 participants, no error bars or significance tests) to carry the load. I ask for (a) a non-3DMM motion metric (e.g., landmark-velocity or optical-flow-based expression/pose distance) or a second 3DMM estimator with a different topology, and (b) a per-item or paired analysis showing that automated motion metrics agree with the user-study 'expression/pose consistency' ratings.
  3. [Sec. 4, Tables 1-8] No error bars, confidence intervals, or significance tests are reported anywhere. Several headline margins are very small: Table 1 self PSNR 21.19 vs 21.13, CSIM 0.879 vs 0.876, LPIPS 0.178 vs 0.187; Table 2 self PSNR 19.98 vs 19.01. With 50 test sequences, these differences may lie within noise. Please report per-metric standard deviations and paired significance tests (or bootstrap CIs) for at least the main comparison tables and for the key ablations in Table 4.
  4. [Sec. 4, Dataset and Evaluation Protocol] The cross-reenactment protocol is under-specified: 'pairs the first frame of one test sequence as the reference identity with a different sequence as the driving video' does not state how many pairs are used, whether all ordered pairs are evaluated, or how the random pairing is fixed. Since cross-reenactment numbers depend strongly on the chosen pairs, please specify the exact evaluation set and report the number of generated videos.
minor comments (5)
  1. [Tables 1, 2, 4, 6, 7] The column 'IQA' is never defined. Please state which image quality assessment is used and how it is computed.
  2. [Table 7] The sentence 'Overall performance improves with scale despite metric-level fluctuations' is not supported by the table: several metrics degrade from 5K to 20K and improve only at 30K, while FID/FVD and IQA fluctuate non-monotonically.
  3. [Table 3] The user-study questionnaire uses a single 'Expression & Pose Consistency' rating. Because the paper's central claim concerns expression and pose separately, please either separate these two questions or justify merging them.
  4. [Sec. 4.2, Table 8] The tracking-quality comparison reports only CSIM/AKD and not AED/APD; adding AED/APD would directly support the sentence 'Accurate tracking is therefore important for identity and geometry control.'
  5. [Sec. 3.3] The autoregressive windowing description says 'nominal overlap of f frames' and 'stride h=s-f', but the pseudo-code/algorithm for tail alignment and pixel-space blending is not given. A short algorithm box or pseudo-code would help reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the derivation is self-contained and benchmarked against external baselines.

full rationale

ViDS's central claim is that dense 3DMM normal maps as conditioning produce more detailed expression/pose control while preserving identity. The derivation chain is a standard supervised fine-tuning setup: Eq. (1) trains a flow-matching velocity field from noisy latents, conditions, and clean latents; Eq. (4) combines classifier-free guidance branches; Eq. (2) defines generation as a function of the source image, normal maps, and prompt. None of these equations defines the predicted video in terms of the evaluation metrics (AED/APD/CSIM), and no evaluation metric is used as a training target. The closest concern is that AED/APD are computed with Deep3DFaceRecon, a 3DMM estimator, while the method conditions on Pixel3DMM/FLAME normal maps. This is an evaluation-confound risk, not a circular reduction: the metrics are computed post hoc on final generated videos by an external estimator, applied identically to all baselines, and the paper acknowledges the limitation ('automated reconstruction can misalign with perceived quality') and supplements with a user study that does not involve 3DMM parameter estimation. The self-citations to Pixel3DMM and SHeaP are used as interchangeable tracking modules with an explicit sensitivity ablation (Table 8), not as authoritative proof of the central claim; no load-bearing argument reduces to an unverified self-citation. The paper is evaluated on external benchmarks (VFHQ, Celeb-V-Text) against external baselines, so the central comparison is not constructed from the method's own assumptions. Accordingly, no circular step meeting the evidence threshold (quote plus specific reduction) is present; the minor self-citation weight is reflected in the score of 1 rather than 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central claim leans on: (i) the accuracy of a self-developed/third-party 3DMM tracker to produce the normal-map control signal; (ii) FLAME's expressiveness to encode desired motion; (iii) the Wan pretrained prior to render geometry into realistic video; and (iv) the validity of 3DMM-derived evaluation metrics. The reader effectively pays for these assumptions upstream; the paper provides ablations supporting (i) and (iii) but no independent ground-truth 3D evaluation for (i), and the AED/APD metrics are coupled to the conditioning modality.

free parameters (2)
  • CFG guidance scales (w_i, w_g, w_p) = 2.0, 1.5, 6.0
    Selected by ablation in Table 6; used in Eq. (4) to balance identity, geometry, and text conditioning. Different values degrade or over-constrain motion.
  • Class-conditioning dropout probability = 0.10
    Set empirically in Sec. 3.2 to enable multiple classifier-free guidance; directly controls how strongly each conditioning branch is learned.
assumptions (4)
  • domain assumption Pixel3DMM [25] provides accurate and stable monocular 3D face reconstruction and expression tracking on the evaluation videos.
    The entire control signal N is rendered from these tracked parameters; Table 8 shows replacing Pixel3DMM with SHeaP degrades results, supporting but not proving accuracy.
  • domain assumption FLAME's parametric shape/expression space is expressive enough to represent the facial motions that matter for portrait animation.
    If FLAME cannot represent some expressions or poses, normal-map conditioning cannot convey them; the paper shows extreme expressions but no coverage analysis.
  • domain assumption The Wan video diffusion prior, after fine-tuning, can synthesize photorealistic faces and follow normal-map conditioning without relying on spurious shortcuts.
    The method assigns motion control to normal maps, but the reference image and text prompt also enter the model; ablations show image/geometry are essential, but cannot prove the model does not also use text or reference cues for motion.
  • domain assumption The evaluation metrics (PSNR/SSIM/LPIPS/CSIM/AED/APD/AKD/FID/FVD/IQA) and the user study are reliable proxies for identity preservation and motion accuracy.
    In particular, AED/APD are computed with Deep3DFaceRecon [18], a 3DMM-based estimator; the paper itself flags misalignment between automated reconstruction and perceived quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViDS: Video Diffusion Shader using 3D Face Tracking." pith.science (2026). https://pith.science/paper/D4EP55EQ

@misc{pith2026260724124,
  author       = {Pith},
  title        = {Pith review of: ViDS: Video Diffusion Shader using 3D Face Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D4EP55EQ}},
  note         = {Machine review of arXiv:2607.24124}
}
read the original abstract

We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/

Figures

Figures reproduced from arXiv: 2607.24124 by the authors.

Figure 1
Figure 1. Given a single face image (first column), our method synthesizes vivid facial animations (second to sixth columns) guided by [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline Overview. We introduce a video diffusion shader for controllable portrait animation from a single image. Given a source image and a driving sequence, the goal is to generate a portrait video that preserves the identity and appearance of the source while following the pose and expression changes in the driving video. We achieve this by using 3DMM tracking as the conditioning signal. We first reconstruct a 3D… view at source ↗
Figure 3
Figure 3. Qualitative comparisons. From top to bottom, we show self-reenactment, cross-reenactment, and in-the-wild generalization. In all three settings, our method outperforms the baselines in handling both extreme head poses and challenging facial expressions, achieving better identity preservation, higher visual quality, and more accurate expression and pose transfer. Additional Evaluation on Celeb-V-Text. To further val￾… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Generation beyond rigid 3DMM constraints. (a) Faithful inner-mouth details, such as teeth and tongue. (b) Text￾prompted hair dynamics. in non-parametric regions. As shown in Fig. 4a and Fig. 4b, our method generates realistic inner-mouth details such as teeth and tongu…
Figure 5
Figure 5. Figure 5: Ablation on conditioning signals. Comparison of dif￾ferent control signals for expression and pose control. etry over a fixed-topology surface, which explains their best overall identity preservation and expression-transfer perfor￾mance in Tab. 4, despite a few metric-…
Figure 6
Figure 6. Figure 6: Ablation on reference shape parameters. Without identity swapping, using the driving subject’s shape parameters produces inaccurate proportions and contours (left). Replacing them with the reference identity parameters restores the correct facial structure (right). Cla…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

86 extracted references · 16 linked inside Pith

  1. [18]

    Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition workshops, pages 0–0, 2019

  2. [1]

    Neural point-based graph- ics

    Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graph- ics. InEuropean conference on computer vision, pages 696–

  3. [2]

    A morphable model for the synthesis of 3d faces

    V olker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. InProceedings of the 26th Annual Conference on Computer Graphics and Interactive Tech- niques, SIGGRAPH 1999, Los Angeles, CA, USA, August 8-13, 1999, pages 187–194. ACM, 1999

  4. [3]

    A groupwise multilinear correspondence optimization for 3d faces

    Timo Bolkart and Stefanie Wuhrer. A groupwise multilinear correspondence optimization for 3d faces. InProceedings of the IEEE international conference on computer vision, pages 3604–3612, 2015

  5. [4]

    A 3d morphable model learnt from 10,000 faces

    James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. A 3d morphable model learnt from 10,000 faces. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5543–5552, 2016

  6. [5]

    3d face morphable models” in-the- wild”

    James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou. 3d face morphable models” in-the- wild”. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 48–57, 2017

  7. [6]

    Multi- linear wavelets: A statistical shape space for human faces

    Alan Brunton, Timo Bolkart, and Stefanie Wuhrer. Multi- linear wavelets: A statistical shape space for human faces. InComputer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceed- ings, Part I 13, pages 297–312. Springer, 2014

  8. [7]

    How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)

    Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). InProceedings of the IEEE international conference on computer vision, pages 1021–1030, 2017

Show all 86 references
  1. [8]

    Facewarehouse: A 3d facial expression database for visual computing.IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013

    Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing.IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013

  2. [9]

    Video- dreamer: Customized multi-subject text-to-video generation with disen-mix finetuning, 2023

    Hong Chen, Xin Wang, Guanning Zeng, Yipeng Zhang, Yuwei Zhou, Feilin Han, and Wenwu Zhu. Video- dreamer: Customized multi-subject text-to-video generation with disen-mix finetuning, 2023

  3. [10]

    Invertible Neural BRDF for Object Inverse Rendering .IEEE Trans- actions on Pattern Analysis & Machine Intelligence, 44(12): 9380–9395, 2022

    Zhe Chen, Shohei Nobuhara, and Ko Nishino. Invertible Neural BRDF for Object Inverse Rendering .IEEE Trans- actions on Pattern Analysis & Machine Intelligence, 44(12): 9380–9395, 2022

  4. [11]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. InPro- ceedings of the AAAI Conference on Artificial Intelligence, pages 2403–2410, 2025

  5. [12]

    Wan-animate: Unified character animation and replacement with holistic replication.arXiv preprint arXiv:2509.14055, 2025

    Gang Cheng, Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Ju Li, Dechao Meng, Jinwei Qi, Penchong Qiao, et al. Wan-animate: Unified character animation and replacement with holistic replication.arXiv preprint arXiv:2509.14055, 2025

  6. [13]

    Generalizable and ani- matable gaussian head avatar.Advances in Neural Informa- tion Processing Systems, 37:57642–57670, 2024

    Xuangeng Chu and Tatsuya Harada. Generalizable and ani- matable gaussian head avatar.Advances in Neural Informa- tion Processing Systems, 37:57642–57670, 2024

  7. [14]

    Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

  8. [15]

    Emoca: Emotion driven monocular face capture and animation

    Radek Dan ˇeˇcek, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 20311–20322, 2022

  9. [16]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 4690–4699, 2019

  10. [17]

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 285–295, 2019

  11. [19]

    Megaportraits: One-shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Alek- sei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022

  12. [20]

    Black, and Timo Bolkart

    Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-wild images.ACM Transactions on Graphics (TOG), 40:1 – 13, 2020

  13. [21]

    Hu- mandit: Pose-guided diffusion transformer for long-form hu- man motion video generation.ArXiv, abs/2502.04847, 2025

    Qijun Gan, Yi Ren, Chen Zhang, Zhenhui Ye, Pan Xie, Xi- ang Yin, Zehuan Yuan, Bingyue Peng, and Jianke Zhu. Hu- mandit: Pose-guided diffusion transformer for long-form hu- man motion video generation.ArXiv, abs/2502.04847, 2025

  14. [22]

    Deferred neural lighting: free-viewpoint re- lighting from unstructured photographs.ACM Trans

    Duan Gao, Guojun Chen, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. Deferred neural lighting: free-viewpoint re- lighting from unstructured photographs.ACM Trans. Graph., 39(6), 2020

  15. [23]

    Learning neural parametric head models

    Simon Giebenhain, Tobias Kirschstein, Markos Georgopou- los, Martin R ¨unz, Lourdes Agapito, and Matthias Nießner. Learning neural parametric head models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21003–21012, 2023

  16. [24]

    Mononphm: Dynamic head reconstruction from monocular videos

    Simon Giebenhain, Tobias Kirschstein, Markos Georgopou- los, Martin R ¨unz, Lourdes Agapito, and Matthias Nießner. Mononphm: Dynamic head reconstruction from monocular videos. InProc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2024

  17. [25]

    Pixel3dmm: Versatile 9 screen-space priors for single-image 3d face reconstruction, 2025

    Simon Giebenhain, Tobias Kirschstein, Martin R ¨unz, Lour- des Agapito, and Matthias Nießner. Pixel3dmm: Versatile 9 screen-space priors for single-image 3d face reconstruction, 2025

  18. [26]

    Liveportrait: Efficient portrait animation with stitching and retargeting control.ArXiv, abs/2407.03168, 2024

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Dingyun Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control.ArXiv, abs/2407.03168, 2024

  19. [27]

    Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis. In2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 5764–

  20. [28]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

  21. [29]

    Belongie, and Ming- Yu Liu

    Zekun Hao, Arun Mallya, Serge J. Belongie, and Ming- Yu Liu. Gancraft: Unsupervised 3d neural rendering of minecraft worlds.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14052–14062, 2021

  22. [30]

    Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022

  23. [31]

    Lam: Large avatar model for one-shot animatable gaus- sian head

    Yisheng He, Xiaodong Gu, Xiaodan Ye, Chao Xu, Zhengyi Zhao, Yuan Dong, Weihao Yuan, Zilong Dong, and Liefeng Bo. Lam: Large avatar model for one-shot animatable gaus- sian head. InProceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Confere...

  24. [32]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

  25. [33]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffu- sion models.ArXiv, abs/2204.03458, 2022. [34]A 3D Face Model for Pose and Illumination Invariant Face Recognition, Genova, Italy, 2009. IEEE

  26. [35]

    3d gaussian splatting for real-time radiance field rendering.ACM Trans

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph., 42(4):139–1, 2023

  27. [36]

    Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023

  28. [37]

    Nersemble: Multi-view ra- diance field reconstruction of human heads.ACM Trans

    Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. Nersemble: Multi-view ra- diance field reconstruction of human heads.ACM Trans. Graph., 42(4), 2023

  29. [38]

    Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and ex- pression from 4D scans.ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):194:1–194:17, 2017

  30. [39]

    Learning a model of facial shape and expression from 4d scans.ACM Trans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans.ACM Trans. Graph., 36(6):194–1, 2017

  31. [40]

    Diffusion- renderer: Neural inverse and forward rendering with video diffusion models

    Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Zhi-Hao Lin, Jun Gao, Alexander Keller, Nan- dita Vijaykumar, Sanja Fidler, and Zian Wang. Diffusion- renderer: Neural inverse and forward rendering with video diffusion models. InThe IEEE Conference on Compu...

  32. [41]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

  33. [42]

    Medi- apipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Medi- apipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019

  34. [43]

    Follow your pose: Pose- guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose- guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 4117–4125, 2024

  35. [44]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. InSIGGRAPH Asia 2024 Conference Papers, pages 1–12, 2024

  36. [45]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021

  37. [46]

    Instant neural graphics primitives with a mul- tiresolution hash encoding.ACM transactions on graphics (TOG), 41(4):1–15, 2022

    Thomas M ¨uller, Alex Evans, Christoph Schied, and Alexan- der Keller. Instant neural graphics primitives with a mul- tiresolution hash encoding.ACM transactions on graphics (TOG), 41(4):1–15, 2022

  38. [47]

    Deepsdf: Learning con- tinuous signed distance functions for shape representation

    Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 165–174, 2019

  39. [48]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 4195–4205, 2023

  40. [49]

    Combining 3d morphable models: A large scale face-and-head model

    Stylianos Ploumpis, Haoyang Wang, Nick Pears, William AP Smith, and Stefanos Zafeiriou. Combining 3d morphable models: A large scale face-and-head model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10934–10943, 2019

  41. [50]

    Joker: Conditional 3d head syn- thesis with extreme facial expressions, 2024

    Malte Prinzler, Egor Zakharov, Vanessa Sklyarova, Berna Kabadayi, and Justus Thies. Joker: Conditional 3d head syn- thesis with extreme facial expressions, 2024

  42. [51]

    3d facial expressions through analysis-by- neural-synthesis

    George Retsinas, Panagiotis P Filntisis, Radek Danecek, Victoria F Abrevaya, Anastasios Roussos, Timo Bolkart, and Petros Maragos. 3d facial expressions through analysis-by- neural-synthesis. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition...

  43. [52]

    High-resolution image 10 synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image 10 synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

  44. [53]

    Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J. Black. Learning to regress 3d face shape and expression from an image without 3d supervision.2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 7755–7764, 2019

  45. [54]

    Sheap: Self-supervised head geometry predictor learned via 2d gaussians.ArXiv, abs/2504.12292, 2025

    Liam Schoneveld, Zhe Chen, Davide Davoli, Jiapeng Tang, Saimon Terazawa, Ko Nishino, and Matthias Nießner. Sheap: Self-supervised head geometry predictor learned via 2d gaussians.ArXiv, abs/2504.12292, 2025

  46. [55]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. InConference on Neural Information Pro- cessing Systems (NeurIPS), 2019

  47. [56]

    Real-time neural radiance talking portrait synthesis via audio-spatial decomposition.CoRR, abs/2211.12368, 2022

    Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition.CoRR, abs/2211.12368, 2022

  48. [57]

    Dphms: Diffusion paramet- ric head models for depth-based tracking

    Jiapeng Tang, Angela Dai, Yinyu Nie, Lev Markhasin, Jus- tus Thies, and Matthias Nießner. Dphms: Diffusion paramet- ric head models for depth-based tracking. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1111–1122, 2024

  49. [58]

    Gaf: Gaussian avatar reconstruction from monocular videos via multi-view diffu- sion

    Jiapeng Tang, Davide Davoli, Tobias Kirschstein, Liam Schoneveld, and Matthias Niessner. Gaf: Gaussian avatar reconstruction from monocular videos via multi-view diffu- sion. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5546–5558, 2025

  50. [59]

    3D face tracking from 2D video through iterative dense UV to image flow

    Felix Taubner, Prashant Raina, Mathieu Tuli, Eu Wern Teh, Chul Lee, and Jinmiao Huang. 3D face tracking from 2D video through iterative dense UV to image flow. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1227–1237, 2024

  51. [60]

    Saragih, Matthias Nießner, Rohit Pandey, S

    Ayush Kumar Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason M. Saragih, Matthias Nießner, Rohit Pandey, S. Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shec...

  52. [61]

    Real- time expression transfer for facial reenactment.ACM Trans

    Justus Thies, Michael Zollh ¨ofer, Matthias Nießner, Levi Val- gaerts, Marc Stamminger, and Christian Theobalt. Real- time expression transfer for facial reenactment.ACM Trans. Graph., 34(6):183–1, 2015

  53. [62]

    Face2face: Real-time face capture and reenactment of rgb videos.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2387–2395, 2016

    Justus Thies, Michael Zollh ¨ofer, Marc Stamminger, Chris- tian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2387–2395, 2016

  54. [63]

    Face2face: Real-time face capture and reenactment of rgb videos

    Justus Thies, Michael Zollhofer, Marc Stamminger, Chris- tian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2387–2395, 2016

  55. [64]

    Headon: Real-time reen- actment of human portrait videos.ACM Transactions on Graphics (TOG), 37(4):1–13, 2018

    Justus Thies, Michael Zollh ¨ofer, Christian Theobalt, Marc Stamminger, and Matthias Nießner. Headon: Real-time reen- actment of human portrait videos.ACM Transactions on Graphics (TOG), 37(4):1–13, 2018

  56. [65]

    De- ferred neural rendering: image synthesis using neural tex- tures.ACM Trans

    Justus Thies, Michael Zollh ¨ofer, and Matthias Nießner. De- ferred neural rendering: image synthesis using neural tex- tures.ACM Trans. Graph., 38(4), 2019

  57. [66]

    Nonlinear 3d face morphable model

    Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7346–7355, 2018

  58. [67]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, P...

  59. [68]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021

  60. [69]

    Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

  61. [70]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024

  62. [71]

    3d face reconstruction with dense landmarks

    Erroll Wood, Tadas Baltru ˇsaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen, Nikola Milosavljevi´c, Daniel Wilde, Stephan Garbin, Toby Sharp, Ivan Stojiljkovi´c, et al. 3d face reconstruction with dense landmarks. InEuropean Confer- ence on Computer Vision, pages 160–177...

  63. [72]

    Vfhq: A high-quality dataset and bench- mark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and bench- mark for video face super-resolution. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 657–666, 2022

  64. [73]

    X-portrait: Expressive portrait anima- tion with hierarchical motion attention

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait anima- tion with hierarchical motion attention. InACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

  65. [74]

    Make- your-video: Customized video generation using textual and structural guidance.arXiv preprint arXiv:2306.00943, 2023

    Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Make- your-video: Customized video generation using textual and structural guidance.arXiv preprint arXiv:2306.00943, 2023

  66. [75]

    11 Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. 11 Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

  67. [76]

    Vasa-1: Lifelike audio-driven talking faces generated in real time.ArXiv, abs/2404.10667, 2024

    Sicheng Xu, Guojun Chen, Yufeng Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time.ArXiv, abs/2404.10667, 2024

  68. [77]

    Hunyuanportrait: Implicit condition con- trol for enhanced portrait animation

    Zunnan Xu, Zhentao Yu, Zixiang Zhou, Jun Zhou, Xiaoyu Jin, Fa-Ting Hong, Xiaozhong Ji, Junwei Zhu, Chengfei Cai, Shiyu Tang, et al. Hunyuanportrait: Implicit condition con- trol for enhanced portrait animation. InProceedings of the Computer Vision and Pattern Recognition Confe...

  69. [78]

    i3dmm: Deep implicit 3d morphable model of human heads

    Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans- Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt. i3dmm: Deep implicit 3d morphable model of human heads. InCVPR, 2021

  70. [79]

    Renderformer: Transformer-based neural rendering of triangle meshes with global illumination

    Chong Zeng, Yue Dong, Pieter Peers, Hongzhi Wu, and Xin Tong. Renderformer: Transformer-based neural rendering of triangle meshes with global illumination. InACM SIG- GRAPH 2025 Conference Papers, 2025

  71. [80]

    Personatalk: Bring attention to your persona in vi- sual dubbing

    Longhao Zhang, Shuang Liang, Zhipeng Ge, and Tianshu Hu. Personatalk: Bring attention to your persona in vi- sual dubbing. InSIGGRAPH Asia 2024 Conference Papers, pages 1–9, 2024

  72. [81]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018

  73. [82]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

  74. [83]

    Zhimeng Zhang, Lincheng Li, and Yu Ding. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset.2021 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 3660– 3669, 2021

  75. [84]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3661–3670, 2021

  76. [85]

    Imface: A nonlinear 3d morphable face model with implicit neural representations

    Mingwu Zheng, Hongyu Yang, Di Huang, and Liming Chen. Imface: A nonlinear 3d morphable face model with implicit neural representations. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20343–20352, 2022

  77. [86]

    Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. InProceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2021

  78. [87]

    Towards metrical reconstruction of human faces

    Wojciech Zielonka, Timo Bolkart, and Justus Thies. Towards metrical reconstruction of human faces. InEuropean Con- ference on Computer Vision, 2022. 12

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.