Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MirrorMe generates real-time, audio-driven half-body portrait animation by compressing video into a diffusion-transformer latent space, reporting 24.33 FPS at 768×512 with strong lip-sync and identity preservation.

desk verdict Coherent architecture and honest ablations, but the SOTA claim rests on baseline numbers that look copied from other papers under different protocols; the paper deserves review with a required unified re-evaluation. read the letter →

arxiv 2506.22065 v1 pith:4AOMHPAY submitted 2025-06-27 cs.CV

Dechao Meng , Steven Xiao , Xindi Zhang , Guangyuan Wang , Peng Zhang , Qi Wang , Bang Zhang , Liefeng Bo This is my paper
classification cs.CV
keywords audio-drivenanimationhalf-bodyportraitdiffusiontransformerlatentvideoreal-timeinferencelip-syncidentitypreservationLTXmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MirrorMe tries to solve the conflict between video quality and speed in audio-driven portrait animation by replacing the usual frame-by-frame diffusion UNet with a video diffusion transformer that denoises the whole clip in a compressed spatiotemporal latent space. The paper argues that this is what makes both temporal coherence and real-time inference possible: with the LTX video model's 1:8192 tokenization, a five-second clip at 24 fps can be generated in about two seconds on a data-center GPU, and the full system runs at 24.33 fps on a consumer-grade GPU. The central claim is that three additions to LTX—reference identity injection by concatenating VAE-encoded image latents, a causal audio encoder and cross-attention adapter aligned to the video VAE's temporal compression, and a progressive training schedule with facial masking and hand-pose conditioning—close the quality gap that heavy compression opens. On the EMTD benchmark, the paper reports better FID, FVD, SSIM, Sync-C, Sync-D, and CSIM than AnimateAnyone, MimicMotion, and EchoMimicV2. If true, this makes interactive audio-driven digital humans practical at consumer speeds rather than batch processing.

What carries the argument

The load-bearing mechanism is the LTX video model's compressed spatiotemporal latent space, a diffusion transformer that maps 32×32×8 pixels per token and compresses time by a factor of 8. On top of that backbone, the causal audio encoder is the component that makes audio control precise: three stacked causal 1D convolutions downsample wav2vec2 frame-level embeddings to the exact temporal resolution of the video latents, and the audio adapter injects local audio features through cross-attention with learnable padding tokens so facial regions respond to speech while backgrounds stay unaffected. The reference identity injection works by concatenating VAE-encoded reference-image latents with noisy video latents in self-attention, with rotary position embedding coordinates adjusted so the reference is spatially aligned but temporally offset, preserving frame-to-frame coherence during long generation.

What would settle it

Re-run AnimateAnyone, MimicMotion, and EchoMimicV2 on the EMTD test videos under the same resolution (768×512), preprocessing, and metric implementations used for MirrorMe, and check whether MirrorMe still leads on Sync-C, CSIM, FID, and FVD by the reported margins. If the gaps shrink to noise or reverse, the claimed state-of-the-art result is an evaluation artifact rather than an architectural advantage.

Watch

Extended reading notes

Core claim

The core discovery is that a video diffusion transformer with aggressive spatiotemporal compression can serve as the backbone for controllable audio-driven animation if the control signals are re-engineered to match its compressed temporal grid. Rather than treating each frame independently, MirrorMe denoises a video latent that represents 32×32×8 pixels per token, so the model is temporally coherent by construction. To keep the reference person's identity, the paper encodes the reference image with the same 3D VAE and concatenates it with the noisy video latents, letting self-attention mix appearance information across the reference and video while rotary position embeddings are offset so the reference does not disturb video-frame ordering. To keep lip-sync under nonuniform temporal compression, the paper stacks three causal 1D convolution layers over wav2vec2 features to compress audio to the video latent's temporal resolution, then adds global audio features to the latents and injects local features through cross-attention with learnable padding tokens. A progressive schedule first trains on cropped faces, then on half-body frames with a dynamic facial mask that confines audio influence to facial regions, plus a face resampling loss and hand keypoint conditioning.

Load-bearing premise

The quantitative comparison assumes that the reported baseline numbers are directly comparable to the authors' own measurements, even though EchoMimicV2 was evaluated at 768×768 while MirrorMe ran at 768×512 and preprocessing and metric versions may differ.

Editorial extensions

If this is right

  • Real-time audio-driven animation at 24 FPS makes conversational avatars and live digital humans plausible on consumer hardware.
  • Because denoising happens in video latent space, generated clips should show less background flicker than image-level diffusion methods.
  • Reference identity can be injected by concatenation with a temporal offset, without retraining the VAE or adding a separate identity encoder.
  • Causal audio compression aligned to the VAE lets one pretrained speech model drive both lip motion and global head and body motion.
  • Motion frames from the previous clip let the system generate arbitrarily long videos by chaining clips with persistent temporal anchors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported speed holds on consumer GPUs, the same compression-plus-control recipe could be applied to other temporal-VAE video models, making real-time controllable generation a property of the backbone rather than a bespoke optimization.
  • The two-stage face-to-halfbody schedule with facial masking is a generic remedy for semantic degradation in heavily compressed video latents; a testable extension is whether it transfers to pose-driven whole-body animation.
  • A direct comparison at matched resolution and metric version is needed before treating the EMTD margins as architectural, since the paper's own numbers show PSNR and hand-quality metrics where pose-constrained baselines remain competitive.
  • The dynamic facial mask implies a tunable trade-off between audio responsiveness and motion blur; varying mask size at inference could produce a controllable expressiveness dial that the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents MirrorMe, an audio-driven half-body animation framework built on the LTX spatiotemporal diffusion transformer. It proposes three components: (i) reference identity injection by concatenating VAE-encoded reference latents with noisy video latents and processing them through self-attention with adjusted RoPE coordinates; (ii) a causal audio encoder and audio adapter to align wav2vec2 features with LTX's temporally compressed video latents; and (iii) a progressive training strategy that first trains on close-up facial animation, then extends to half-body synthesis with a facial mask and hand-pose guidance, together with a masked face velocity loss (Eq. 8). Evaluation on the EMTD benchmark reports improved FID, FVD, SSIM, SyncNet, and CSIM over AnimateAnyone, MimicMotion, and EchoMimicV2, as well as real-time inference at 24.33 FPS at 768x512 resolution. An ablation study decomposes the contribution of each proposed module.

Significance. If the quantitative comparisons are valid, MirrorMe would be a meaningful advance: it demonstrates a diffusion-transformer architecture that achieves real-time audio-driven half-body animation with temporal coherence, and the design rationale for each module is clearly motivated by LTX's spatiotemporal compression. The ablation rows are internally consistent with the claimed contributions, and the real-time inference result is genuinely interesting in a literature dominated by slower UNet-based methods. However, the headline state-of-the-art conclusion is currently undermined by the uncontrolled baseline comparison, the absence of any statistical reliability measures, and incomplete details in the speed comparison. The underlying architecture and ablation are still worthy of publication after the comparison evidence is substantiated or the claims are appropriately narrowed.

major comments (3)
  1. [Section 4.2, Table 1] The central claim of state-of-the-art performance rests on metrics that are not established as comparable across methods. The paper does not state that AnimateAnyone, MimicMotion, and EchoMimicV2 were re-run under MirrorMe's evaluation protocol; the table's EchoMimicV2 row (Sync-C 7.219, HKV 25.28) is essentially identical to the values in the EchoMimicV2 paper, which suggests the numbers were copied rather than recomputed. No evaluation details are given for reference-frame selection, audio length, SyncNet version and preprocessing, FVD implementation, or hand-keypoint extractor, and the output resolutions differ (EchoMimicV2 at 768x768 vs. MirrorMe at 768x512). Without a unified protocol or at least a documented re-evaluation, the reported margins over EchoMimicV2 (FID 49.33 vs. 43.53; Sync-C 7.219 vs. 7.232) cannot be attributed to the method. Moreover, only one of the three competitors (EchoMimicV2) is actually audio-driven for half-body animation; AnimateAnyone is pose-driven, and strong audio-driven baselines such as CyberHost and Vlogger are omitted from the quantitative table. The ablation rows are internally informative, but they do not establish cross-method SOTA by themselves.
  2. [Section 4.2, Table 1] No error bars, confidence intervals, or statistical tests are reported for any metric. FID, FVD, SSIM, and SyncNet scores are single numbers, despite the stochastic nature of diffusion sampling and the known sensitivity of FVD/FID to implementation details. The differences between MirrorMe and EchoMimicV2 on Sync-C (7.232 vs. 7.219) and HKV (25.32 vs. 25.28) are far smaller than typical run-to-run variation; these particular margins are effectively tie-level. At minimum, multiple seeds or bootstrapped confidence intervals are needed before a SOTA claim can be made.
  3. [Section 4.6, Table 2] The real-time claim is not fully substantiated. The table reports 24.33 FPS for MirrorMe at 768x512 versus 0.32/3.20 FPS for EchoMimicV2 at 768x768, but it omits the GPU model, sampling steps, batch size, and whether the baseline speeds were measured on the same machine. The text is inconsistent about hardware: Section 3.1 mentions H100 GPUs, while Section 4.6 refers to 'consumer-grade NVIDIA GPUs.' Resolution also differs, so the claimed 8x speedup 'at the same inference resolution and diffusion steps' is not supported by the table as presented.
minor comments (7)
  1. [Section 3.4] The text references 'Figure 5(b)' for the progressive training pipeline, but the illustration appears in Figure 3(b); the cross-reference is incorrect.
  2. [Equation (7)] The scalar gamma in Equation (7) is introduced without specifying whether it is learned, fixed, or schedule-dependent; please clarify.
  3. [Section 3.4, Eq. (8)] The loss in Eq. (8) is called the 'face resampling loss,' but it is an L2 loss on masked latent velocities; the terminology should be aligned with the actual mechanism, or the mechanism should be explained in more detail.
  4. [Section 4.6, Table 2] The hardware and sampling configuration for the FPS measurements are not reported, and the table header spells the baseline inconsistently as 'EchoMimic2' rather than EchoMimicV2.
  5. [Section 4.4] The qualitative comparison is performed on demos from CyberHost's project page rather than on a fixed, independently selected test set; this does not support a claim of superiority over CyberHost, which is not included in the quantitative comparison.
  6. [Section 4.2] HKV is described as measuring the 'diversity of hand movements,' but higher HKV is not self-evidently better unless hand-pose accuracy is also verified, since uncontrolled jitter could inflate variance; consider reporting a pose-accuracy metric alongside HKV.
  7. [References] References [36] and [37] cite the same EMO paper with different venue years, and reference [38] contains malformed author names; please correct these entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MirrorMe's architecture, losses, and ablations are self-contained; the SOTA comparison raises protocol-fairness concerns, not a circular reduction.

full rationale

The paper's derivation chain is self-contained: the reference identity injection, causal audio encoder, audio adapter, and progressive training are all described with concrete architectural components and losses. The face resampling loss (Eq. 8) supervises facial latent regions with an L2 loss against ground-truth latents, and the ablation rows in Table 1 show incremental Sync-C/Sync-D changes when modules are added; SyncNet is an external metric applied uniformly, so the improvement is an empirical outcome rather than an identity or a renamed fit. The base model LTX is external prior work, the audio encoder starts from pretrained wav2vec2, and the pose conditioning uses externally extracted hand keypoints; no load-bearing claim is justified solely by a self-citation. Although AnimateAnyone shares authors with MirrorMe, its Table 1 row functions as an external baseline and the central architecture does not depend on that citation. The only substantive concern is that Table 1 lists metrics for AnimateAnyone, MimicMotion, and EchoMimicV2 without stating whether they were re-measured under MirrorMe's protocol; EchoMimicV2's published values closely track the Table 1 entries, and EchoMimicV2 was evaluated at 768x768 while MirrorMe uses 768x512. That is an evaluation-fairness and correctness risk, not circularity under the definitions used here, because no fitted parameter is relabeled as a prediction and no equation reduces to its own input. Under the review rules, protocol-mismatch concerns belong in correctness risk, not in the circularity score.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain assumptions about the base model (LTX) and the evaluation metrics, plus several model and training hyperparameters chosen by hand. No new physical or ontological entities are introduced; the audio adapter and pose encoder are architectural modules, not free-standing entities, so they are not listed as invented entities.

free parameters (6)
  • face loss weight λ
    Hyperparameter in Ltotal = LMSE + λ Lface (Eq. 9); value not reported; chosen by hand.
  • guidance scale = 2.5
    Classifier-free guidance scale for inference; set manually.
  • timestep shift coefficient = 17
    Shifts timestep sampling during training; chosen heuristically.
  • control signal dropout = 10%
    Independent dropout probability for reference image, audio, and hand poses during training.
  • adapter insertion interval k
    Audio adapter is applied every k transformer blocks; k is not specified in the text.
  • training steps per phase = 10k / 30k
    First phase 10,000 steps, second phase 30,000 steps; schedule chosen without reported tuning.
assumptions (5)
  • domain assumption The LTX video model's 3D VAE and diffusion transformer provide a valid, high-compression latent space for video generation.
    The entire method depends on LTX's 1:8192 spatiotemporal compression retaining enough semantic detail; this is assumed, not proven here.
  • domain assumption wav2vec2 features contain sufficient phonetic information for lip-sync.
    Used as the audio backbone in Section 3.3; the paper does not evaluate alternative audio encoders.
  • domain assumption SyncNet (Sync-C, Sync-D) is an accurate measure of audio-visual synchronization.
    Used as the primary lip-sync metric in Section 4.2; no validation against human perception is given.
  • domain assumption The face masks Mface used in Eq. 8 are accurate pixel-wise masks of facial regions.
    The quality of the face loss depends on mask precision; the mask extraction method is not specified.
  • domain assumption The collected 10,000 clips (200+ hours) are representative and diverse enough to train a generalizable model.
    Data collection is not described in sufficient detail to assess bias or coverage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation." pith.science (2026). https://pith.science/paper/4AOMHPAY

@misc{pith2026250622065,
  author       = {Pith},
  title        = {Pith review of: MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AOMHPAY}},
  note         = {Machine review of arXiv:2506.22065}
}
read the original abstract

Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX's trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX's temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.

Figures

Figures reproduced from arXiv: 2506.22065 by the authors.

Figure 1
Figure 1. MirrorMe is a DiT-base audio-driven portrait animation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of MirrorMe. During training, the reference image is encoded through the VAE, and the encoded features [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of (a) causal audio encoder and (b) progres [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cross-Identity Synthesis Results Driven by Multimodal [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The qualitative results of MirrorMe compare to other audio-driven halfbody animation methods. Some local areas were enlarged [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the temporal consistency of our model [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Live Avatar reports real-time streamable generation from a 14B audio-driven diffusion model at ~20 FPS on 5 H800s with stable identity over 10,000 seconds.

Reference graph

Works this paper leans on

51 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020

    Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020. 5

    work page 2020

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. 1, 2

    2023

  3. [3]

    Vlogger: Multimodal diffusion for embodied avatar synthe- sis

    Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthe- sis. arXiv preprint arXiv:2403.08764, 2024. 3

    arXiv 2024

  4. [4]

    Headgan: One-shot neural head synthesis and editing

    Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In Proceedings of the IEEE/CVF International con- ference on Computer Vision, pages 14398–14407, 2021. 2

    work page 2021

  5. [5]

    Emoportraits: Emotion-enhanced multimodal one-shot head avatars

    Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos V ougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pan- tic. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8498– 8507, 2024. 2

    work page 2024

  6. [6]

    Scaling rectified flow trans- formers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yan- nik Marek, and Robin Rombach. Scaling rectified flow trans- formers for high-resolution image synthesis, 2024. 3

    work page 2024

  7. [7]

    High-fidelity and freely controllable talking head video generation

    Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 5609–5619, 2023. 1

    work page 2023

  8. [8]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 1, 2

    2020

Show all 51 references
  1. [9]

    Densepose: Dense human pose estimation in the wild

    Rıza Alp G ¨uler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) , pages 7297–7306,

  2. [10]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168, 2024. 1, 2

    2024 arXiv

  3. [11]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning, 2024

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning, 2024. 1, 2

    2024

  4. [12]

    Ltx-video: Realtime video latent diffusion, 2024

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weiss- buch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion,...

    2024

  5. [13]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010. 6

    2010

  6. [14]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation, 2023

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation, 2023. 3, 6, 8

    2023

  7. [15]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024. 3

    2024 arXiv

  8. [16]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 2

    2013 arXiv

  9. [17]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 2

    2024 arXiv

  10. [18]

    Cyberhost: Taming audio- driven avatar diffusion model with region codebook atten- tion, 2024

    Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Ji- aqi Yang, and Yanbo Zheng. Cyberhost: Taming audio- driven avatar diffusion model with region codebook atten- tion, 2024. 3, 7

    2024

  11. [19]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxi- milian Nickel, and Matt Le. Flow matching for generative modeling, 2023. 3

    2023

  12. [20]

    Contextual gesture: Co- speech gesture video generation through context-aware ges- ture representation

    Pinxin Liu, Pengfei Zhang, Hyeongwoo Kim, Pablo Garrido, Ari Sharpio, and Kyle Olszewski. Contextual gesture: Co- speech gesture video generation through context-aware ges- ture representation. arXiv preprint arXiv:2502.07239, 2025. 1

    2025

  13. [21]

    Dreamtalk: When emotional talking head generation meets diffusion probabilistic mod- els

    Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When emotional talking head generation meets diffusion probabilistic mod- els. arXiv preprint arXiv:2312.09767, 2023. 1

    2023 arXiv

  14. [22]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–12, 2024. 3

    2024

  15. [23]

    Echomimicv2: Towards striking, simplified, and semi-body human animation, 2024

    Rang Meng, Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimicv2: Towards striking, simplified, and semi-body human animation, 2024. 2, 3, 6, 7, 8

    2024

  16. [24]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 3

    2023

  17. [25]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  18. [26]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia , pages 484–492, 2020. 6

    2020

  19. [27]

    Pirenderer: Controllable portrait image generation via semantic neural rendering

    Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In Proceedings of the IEEE/CVF international conference on computer vision , pages 13759– 13768, 2021. 1

    2021

  20. [28]

    High-resolution image syn- thesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2022. 2

    2022

  21. [29]

    wav2vec: Unsupervised pre-training for speech recognition, 2019

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition, 2019. 5

    2019

  22. [30]

    Difftalk: Crafting diffusion models for generalized audio-driven portraits animation

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 1982–1991, 2023. 1

    1982

  23. [31]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019. 2

    2019

  24. [32]

    Motion Representations for Ar- ticulated Animation

    Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion Representations for Ar- ticulated Animation. 2021. 2

    2021

  25. [33]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  26. [34]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. 4

    2023

  27. [35]

    Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior

    Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841, 2023. 1, 2

    2023 arXiv

  28. [36]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pages 244–260. Springer, 2024. 1

    2024

  29. [37]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pages 244–260. Springer, 2025. 3

    2025

  30. [38]

    Video generation models as world simulators

    Brooks Tim, Peebles Bill, Connorm Holmes, DePue Will, Yufeim Guo, Jing Li, Schnurr David, Taylor Joe, Luhman Troy, Luhman Eric, Ng Clarence, Wang Ricky, and Ramesh Aditya. Video generation models as world simulators. 2024. Accessed: 2024-02-15. 2

    2024

  31. [39]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  32. [40]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 6

    2004

  33. [41]

    Easyanimate: A high-performance long video generation method based on transformer architecture

    Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture. arXiv preprint arXiv:2405.18991,

  34. [42]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 1

    2024 arXiv

  35. [43]

    Vasa-1: Lifelike audio-driven talking faces generated in real time

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. Advances in Neural Information Pro- cessing Systems, 37:660–684, 2025. 1, 2

    2025

  36. [44]

    Magicanimate: Temporally consistent human image animation using diffusion model, 2023

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model, 2023. 3

    2023

  37. [45]

    Cogvideox: Text-to-video diffusion models with an expert transformer, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models ...

    2024

  38. [46]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2

    2024 arXiv

  39. [47]

    Metaportrait: Identity-preserving talking head gener- ation with fast personalized adaptation

    Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, Hsiang- Tao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: Identity-preserving talking head gener- ation with fast personalized adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2023

  40. [48]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  41. [49]

    Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance. arXiv preprint arXiv:2406.19680, 2024. 7, 8

    2024 arXiv

  42. [50]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 2

    2024

  43. [51]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 1

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.