Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A diffusion-based audio-driven portrait video system can generate an interactive avatar in real time, sustaining 78 FPS at 384x384 with a 140 ms initial response delay on an RTX 4090D.

desk verdict Plausible latency numbers, honest engineering; the fidelity claims need quantitative backup before the 140 ms interaction story can be trusted. read the letter →

arxiv 2506.05806 v1 pith:JIAFUH4Z submitted 2025-06-06 cs.CV

Haojie Yu , Zhaonian Wang , Yihan Pan , Meng Cheng , Hao Yang , Chao Wang , Tao Xie , Xiaoming Xu
show 2 more authors
Xiaoming Wei Xunliang Cai
This is my paper
classification cs.CV
keywords audio-drivenportraitvideogenerationreal-timediffusioninferenceconsistencymodeldistillationlow-latencyinteractiveavatarsvariable-lengthquantizationfacialexpressioncontroltalkingheadanimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that diffusion models do not have to be excluded from real-time interactive avatar applications: a carefully engineered audio-to-portrait pipeline can generate talking-head video at interactive latency and frame rate while preserving fidelity. The core move is to attack response latency directly—training the model to generate very short initial clips ($f=4$ frames) so the user's first animated reply appears in roughly 140 ms—rather than treating steady-state generation speed as the only bottleneck. If the measurements on an RTX 4090D hold, diffusion renderers could replace GAN-based renderers in interactive settings, gaining expressiveness without sacrificing interactivity. The paper also adds conversational state control (speaking, listening, idle) via class labels and explicit facial-expression control, both aimed at making two-way interaction feel natural.

What carries the argument

The load-bearing mechanisms are three. (1) Variable-length video generation training: early training keeps the generated clip length $f$ fixed so the motion module and audio layers learn well; later training makes $f$ dynamic with $f\ge n$ (motion frames), so the model can generate short clips such as $f=4$ at inference without accumulating error—this is what cuts response latency to 140 ms. (2) Consistency-model distillation for audio-to-video: a DDPM-pretrained UNet is trained with a latent consistency loss (using a DDIM solver for PF-ODE point pairs) plus an audio-conditioned, reference-conditioned discriminator, enabling 2-step sampling; temporal flicker is reduced by a frame-weighting smoothing loss. (3) Deployment engineering: TensorRT, INT8 quantization of the UNet with sensitive layers kept in high precision, and pipeline parallelism that runs the UNet and VAE concurrently. Supporting the interaction design, class labels are added to the timestep embedding, and expression control edits the reference image with a portrait-animation model before feeding it to ReferenceNet.

What would settle it

Render the same identity and audio under the $f=4$ and $f=12$ clip lengths and under 2-step versus 25-step sampling, then measure lip-sync error (e.g., LSE-D), identity similarity, and temporal consistency on a held-out set. If the short-clip or 2-step variants are measurably worse in the first few seconds, the 140 ms latency is achieved by shipping lower-fidelity video. A complementary check is to extend a session over many generated segments and measure drift, since the dynamic-length training claim is specifically that error does not accumulate for lengths outside the training range.

Watch

Extended reading notes

Core claim

The paper's central claim is that a diffusion-based audio-image-to-video model, trained with dynamic variable-length video generation and consistency-model distillation, can run in real time at interactive latency: 78 FPS at $384\times384$ resolution with a 140 ms initial response delay, and 45 FPS at $512\times512$ with 215 ms, on a single RTX 4090D GPU. The quality of the generated talking head is claimed to remain comparable to 25-step sampling when only 2 or 4 sampling steps are used, and the dynamic-length training is claimed to prevent the error accumulation that normally appears when a diffusion video model generates clips shorter than its training length. The paper further claims that class labels embedded in the timestep conditioning let the avatar switch between speaking, listening, and idle states, and that facial expressions can be edited throughout a video by animating the reference image before it enters the appearance network.

Load-bearing premise

The whole latency-versus-quality case rests on the assumption that a 4-frame initial clip and a 2-step distilled model preserve lip-sync, identity, and motion coherence as well as longer, many-step generation—an assumption the paper supports only with a qualitative figure and no metrics for these settings.

Editorial extensions

If this is right

  • At 384x384, the model sustains up to 78 FPS with a 140 ms initial response delay on an RTX 4090D, meeting real-time playback on a single consumer-grade GPU.
  • At 512x512, it sustains 45 FPS with a 215 ms initial delay, still within real-time territory.
  • Response latency is decoupled from steady-state frame rate: the first clip is only 4 frames long, while later segments use 12 frames for higher quality, so the user sees a quick first reply without sacrificing long-clip coherence.
  • Two-step and four-step consistency-distilled sampling are claimed to be visually close to 25-step sampling, making few-step diffusion practical for audio-driven portraits.
  • Class labels allow seamless switching among speaking, listening (with nods/smiles), and idle states, and expression templates can be switched every few seconds across a 20-second video.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 140 ms covers only the video-generation modules (pipeline plus VAE decoder); the end-to-end wait a user experiences includes speech recognition, large-language-model response, text-to-speech, and audio encoding, so the true interactive latency is longer than 140 ms.
  • If the dynamic variable-length claim holds, the same training trick should transfer to other streaming diffusion video generators—anything that needs to emit the first chunk of video quickly and continue without drift—because the bottleneck it attacks is structural, not specific to talking heads.
  • The expression-control design—editing the reference image before it enters the appearance network—exploits a known shortcut in reference-based animators (the model copies reference expression), so it is likely portable to other ReferenceNet-style animators at near-zero inference cost.
  • A testable extension is to push resolution or refresh rate further on GPUs that support FP8 or with the audio encoder also quantized; conversely, on GPUs weaker than an RTX 4090D the real-time claim needs re-measurement before being treated as portable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents LLIA, a diffusion-based audio-driven portrait video generation system designed for real-time interactive avatars. The main contributions are: (1) a variable-length video generation training strategy that allows short initial clips (f=4) to reduce response latency; (2) consistency-model training (LCM plus adversarial losses) for few-step (2-4 step) inference; (3) INT8 quantization and pipeline parallelism for throughput; (4) class-label conditioned state switching (speaking/listening/idle); and (5) fine-grained facial expression control by editing the reference image with a portrait animation model. The paper reports 78 FPS at 384×384 and 45 FPS at 512×512 on an RTX 4090D, with initial latencies of 140 ms and 215 ms. Quality and interaction assessments are presented through qualitative examples; no quantitative quality metrics or comparisons to prior work are provided.

Significance. If the claims hold, the system would be a strong engineering demonstration that diffusion-based portrait animation can achieve interactive latency and streaming frame rates. The paper correctly identifies response latency as a key but under-addressed issue in interactive avatar systems, and the combination of variable-length training, few-step distillation, quantization, and pipeline parallelism is a sensible design. The work is potentially valuable to practitioners building interactive portrait avatars. However, its impact is currently limited by the absence of quantitative quality evidence at the claimed operating points and the lack of comparison with existing systems.

major comments (4)
  1. [§4.3, Table 2] Table 2 reports lower inference times for 25-step sampling than for 4-step sampling at the same resolution (e.g., 25-step 512×512/4 rows: pipe 147 ms vs 4-step 233 ms; 25-step 512×512/12: 265 ms vs 469 ms). Since denoising time should increase with the number of sampling steps, these entries are inconsistent and suggest the first column is mislabeled (likely '2' instead of '25'). The speed claims in Section 4.3 are built on this table, so the table must be corrected and the measurement protocol clarified.
  2. [§4.4, §4.3] The headline 78 FPS / 140 ms figures are obtained with 2-step sampling and an initial clip of f=4 frames, but no quantitative evaluation of image quality, lip-sync accuracy, identity preservation, or temporal consistency is reported for these configurations. Figure 4 is a single qualitative comparison and the statement that 2-step and 4-step show 'virtually no difference' is not supported by any metric. This is load-bearing for the paper's central claim that real-time speed is achieved while maintaining high-fidelity output; without such measurements, the latency numbers could reflect seriously degraded video.
  3. [Abstract, §1 third bullet, §4] The abstract and contributions list a 'long-term stable inference strategy' for very few-step sampling, but the body of the paper never describes this strategy. Section 4.3 only mentions choosing f=4 for initial segments and f=12 later; the 'specific strategy' for guiding idle-state diffusion to predefined states is absent. The sustained-interaction claim is therefore unverifiable.
  4. [§4.3, §4.4] The experiments contain no comparison with existing audio-driven portrait generation methods (e.g., EMO, Hallo, VASA-1, EchoMimic) on either quality or speed. Without baselines, the claim of 'extensive experiments' demonstrating state-of-the-art-like performance is unsubstantiated. I would expect at least a few standard metrics (e.g., LSE-C/LSE-D, identity similarity, FID/LPIPS) or a user study on a common benchmark, in addition to the latency numbers.
minor comments (4)
  1. [Table 2 caption] The notation 'pipe' is not defined in the caption; define it in the caption, since the text only explains it in Section 4.3.
  2. [§4.1, Reference [36]] There is a typo in Section 4.1: 'V oxCeleb2' should be 'VoxCeleb2', and reference [36] has 'Augest' instead of 'August'.
  3. [Figure 4 caption] The caption of Figure 4 does not state which resolution, sampling steps, or clip length were used for the comparison; these details are needed to interpret the qualitative result.
  4. [Eq. (2)-(3)] The meaning of the subscripts in Equations (2) and (3) and the definition of the weighting w are not provided; a short notation description would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the latency/FPS numbers are measured per-module runtimes, and no claimed prediction is fitted to or derived from its own inputs.

full rationale

The paper's central engineering claims are speed and latency, and these are reported as measured per-module timings (Table 1, Table 2, Section 4.3). The 140 ms response latency is computed as 'pipe' speed plus 40 ms for the VAE decoder, and the FPS values are derived from inference measurements, not from any fitted parameter or self-referential definition. The variable-length training strategy, consistency-model training, quantization, and pipeline parallelism are presented as empirical engineering contributions; none of them defines a prediction in terms of a fitted output. The quality claims (Section 4.4) rest on the authors' own qualitative figures and a self-built dataset, and no quantitative lip-sync, identity, or temporal-consistency metric is reported for the f=4 or 2-step operating points used in the headline latency numbers. That is an evaluation-influence and completeness concern, not circularity: the system is not validated against an external benchmark, but the speed derivation is self-contained arithmetic over measured runtimes. There is one minor self-citation in the data-filtering pipeline (YOLOv6, reference [39], with overlapping authors), but it is used only as a face-detection tool and is not load-bearing for any claimed result. An internal consistency oddity exists (78 FPS at 384x384 does not follow from 140 ms for a 4-frame clip, 4/0.140 ≈ 28.6 FPS, unless the FPS number refers to a different configuration), but this is an arithmetic or reporting issue, not a circular derivation. No step in the paper reduces, by the paper's own equations or by self-citation, to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central latency claim depends mainly on hardware and measurement assumptions rather than on mathematical axioms. The quality claim depends on several domain assumptions inherited from pretrained backbones and from the self-curated dataset. No new physical or conceptual entities are introduced.

free parameters (3)
  • Loss weights lambda_0 and lambda_1 = not reported
    In Eq. 5 L = lambda_0 * L_lcm + lambda_1 * L_G, the balance between LCM and adversarial losses is not specified, though it determines the quality-speed trade-off.
  • Temporal consistency loss weight = 0.15
    A 'supplementary loss between frames (weight = 0.15)' is introduced in extra Stage II; this hand-set weight affects temporal quality claims.
  • Dynamic generation length f = 4 to 12 frames
    The minimum initial clip length f=4 is chosen for latency, later f=12 for quality; the choice is empirical and directly controls the advertised 140 ms and 215 ms latency.
assumptions (4)
  • domain assumption Pretrained SD1.5 and AnimateDiff v2 motion module provide a strong enough prior for identity and motion after fine-tuning.
    Section 4.2 initializes weights from SD1.5 and AnimateDiff; the method inherits any biases or limitations of these backbones.
  • standard math The LCM objective with DDIM PF-ODE pairing yields valid consistency targets for this multi-condition task.
    Section 3.5 adopts latent consistency models and uses the DDIM solver to construct adjacent point pairs; the theory is assumed from reference [35].
  • domain assumption Portrait-animation-modified references can serve as expression conditioning without breaking identity.
    Section 3.4 relies on the ReferenceNet copying shortcut and a portrait animation model to change expression in the reference image; no identity preservation metric is reported.
  • domain assumption The web-curated and synthetic dataset is representative of interactive state transitions.
    Section 4.1 describes 171 and 158 hours of filtered web data plus 4 hours of synthetic listening and idle data; distribution coverage is asserted, not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models." pith.science (2026). https://pith.science/paper/JIAFUH4Z

@misc{pith2026250605806,
  author       = {Pith},
  title        = {Pith review of: LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIAFUH4Z}},
  note         = {Machine review of arXiv:2506.05806}
}
read the original abstract

Diffusion-based models have gained wide adoption in the virtual human generation due to their outstanding expressiveness. However, their substantial computational requirements have constrained their deployment in real-time interactive avatar applications, where stringent speed, latency, and duration requirements are paramount. We present a novel audio-driven portrait video generation framework based on the diffusion model to address these challenges. Firstly, we propose robust variable-length video generation to reduce the minimum time required to generate the initial video clip or state transitions, which significantly enhances the user experience. Secondly, we propose a consistency model training strategy for Audio-Image-to-Video to ensure real-time performance, enabling a fast few-step generation. Model quantization and pipeline parallelism are further employed to accelerate the inference speed. To mitigate the stability loss incurred by the diffusion process and model quantization, we introduce a new inference strategy tailored for long-duration video generation. These methods ensure real-time performance and low latency while maintaining high-fidelity output. Thirdly, we incorporate class labels as a conditional input to seamlessly switch between speaking, listening, and idle states. Lastly, we design a novel mechanism for fine-grained facial expression control to exploit our model's inherent capacity. Extensive experiments demonstrate that our approach achieves low-latency, fluid, and authentic two-way communication. On an NVIDIA RTX 4090D, our model achieves a maximum of 78 FPS at a resolution of 384x384 and 45 FPS at a resolution of 512x512, with an initial video generation latency of 140 ms and 215 ms, respectively.

Figures

Figures reproduced from arXiv: 2506.05806 by the authors.

Figure 1
Figure 1. Overview of the proposed method. Compared with previous work, our pipeline introduces several novel [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Illustration of Inference Pipeline Parallelism. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The qualitative results based on different avatar styles. The left image is the reference image. We sample four [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: From left to right, the portraits are generated with the same audio segment under 2-step, 4-step, and 25-step [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the digital avatar in the listening state. The avatar performs a nodding response action according [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The qualitative results of expression control. In the generated 20-second video, the model smoothly switches [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EchoCache: Energy-Guided Cross-Modal Caching for Efficient Audio-Driven Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Audio time-frequency energy guides which video latents get recomputed during diffusion denoising, yielding up to 2.46x faster audio-driven video generation with competitive quality.

  2. Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Live Avatar reports real-time streamable generation from a 14B audio-driven diffusion model at ~20 FPS on 5 H800s with stable identity over 10,000 seconds.

Reference graph

Works this paper leans on

40 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. InProceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020

    2020

  2. [2]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. InEuropean Conference on Computer Vision, pages 244–260. Springer, 2024

    2024

  3. [3]

    Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Processing Systems, 37:660–684, 2024

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Processing Systems, 37:660–684, 2024

    2024

  4. [4]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8652–8661, 2023. 10

    work page 2023

  5. [5]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    arXiv 2010

  6. [6]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  7. [7]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  8. [8]

    Megaportraits: One-shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. InProceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022

    work page 2022

Show all 40 references
  1. [9]

    Infp: Audio-driven interactive head generation in dyadic conversations.arXiv preprint arXiv:2412.04037, 2024

    Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, and Zhipeng Ge. Infp: Audio-driven interactive head generation in dyadic conversations.arXiv preprint arXiv:2412.04037, 2024

    2024 arXiv

  2. [10]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2403–2410, 2025

    2025

  3. [11]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    2024 arXiv

  4. [12]

    Hallo2: Long-duration and high-resolution audio-driven portrait image animation.arXiv preprint arXiv:2410.07718, 2024

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait image animation.arXiv preprint arXiv:2410.07718, 2024

    2024 arXiv

  5. [13]

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer.arXiv preprint arXiv:2412.00733, 2024

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer.arXiv preprint arXiv:2412.00733, 2024

    2024 arXiv

  6. [14]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation.arXiv preprint arXiv:2403.17694, 2024

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation.arXiv preprint arXiv:2403.17694, 2024

    2024 arXiv

  7. [15]

    Cyberhost: Taming audio-driven avatar diffusion model with region codebook attention.arXiv preprint arXiv:2409.01876, 2024

    Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, and Yanbo Zheng. Cyberhost: Taming audio-driven avatar diffusion model with region codebook attention.arXiv preprint arXiv:2409.01876, 2024

    2024 arXiv

  8. [16]

    Loopy: Taming audio- driven portrait avatar with long-term motion dependency.arXiv preprint arXiv:2409.02634, 2024

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio- driven portrait avatar with long-term motion dependency.arXiv preprint arXiv:2409.02634, 2024

    2024 arXiv

  9. [17]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023

    2023

  10. [18]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024

    2024

  11. [19]

    Liveportrait: Efficient portrait animation with stitching and retargeting control.arXiv preprint arXiv:2407.03168, 2024

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control.arXiv preprint arXiv:2407.03168, 2024

    2024 arXiv

  12. [20]

    X-portrait: Expressive portrait animation with hierarchical motion attention

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait animation with hierarchical motion attention. InACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  13. [21]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  14. [22]

    First order motion model for image animation.Advances in neural information processing systems, 32, 2019

    Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation.Advances in neural information processing systems, 32, 2019

    2019

  15. [23]

    Omnitalker: Real-time text-driven talking head generation with in-context audio-visual style replication.arXiv preprint arXiv:2504.02433, 2025

    Zhongjian Wang, Peng Zhang, Jinwei Qi, Guangyuan Wang Sheng Xu, Bang Zhang, and Liefeng Bo. Omnitalker: Real-time text-driven talking head generation with in-context audio-visual style replication.arXiv preprint arXiv:2504.02433, 2025

    2025 arXiv

  16. [24]

    Chatanyone: Stylized real-time portrait video generation with hierarchical motion diffusion model.arXiv preprint arXiv:2503.21144, 2025

    Jinwei Qi, Chaonan Ji, Sheng Xu, Peng Zhang, Bang Zhang, and Liefeng Bo. Chatanyone: Stylized real-time portrait video generation with hierarchical motion diffusion model.arXiv preprint arXiv:2503.21144, 2025

    2025 arXiv

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  18. [26]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  19. [27]

    wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural information processing systems, 33:12449– 12460, 2020

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural information processing systems, 33:12449– 12460, 2020

    2020

  20. [28]

    wav2vec: Unsupervised pre-training for speech recognition.arXiv preprint arXiv:1904.05862, 2019

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition.arXiv preprint arXiv:1904.05862, 2019

    1904 arXiv

  21. [29]

    chinese speech pretrain, 2022

    Pengcheng Guo and Shixing Liu. chinese speech pretrain, 2022

    2022

  22. [30]

    Hsemotion: High-speed emotion recognition library.Software Impacts, 14:100433, 2022

    Andrey V Savchenko. Hsemotion: High-speed emotion recognition library.Software Impacts, 14:100433, 2022

    2022

  23. [31]

    Animatediff-lightning: Cross-model diffusion distillation.arXiv preprint arXiv:2403.12706, 2024

    Shanchuan Lin and Xiao Yang. Animatediff-lightning: Cross-model diffusion distillation.arXiv preprint arXiv:2403.12706, 2024

    2024 arXiv

  24. [32]

    Sdxl-lightning: Progressive adversarial diffusion distillation.arXiv preprint arXiv:2402.13929, 2024

    Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation.arXiv preprint arXiv:2402.13929, 2024

    2024 arXiv

  25. [33]

    Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    2020

  26. [34]

    Animatelcm: Computation-efficient personalized style video generation without personalized video data

    Fu-Yun Wang, Zhaoyang Huang, Weikang Bian, Xiaoyu Shi, Keqiang Sun, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Computation-efficient personalized style video generation without personalized video data. In SIGGRAPH Asia 2024 Technical Communications, pages 1–5. 2024

    2024

  27. [35]

    Latent consistency models: Synthesizing high-resolution images with few-step inference.arXiv preprint arXiv:2310.04378, 2023

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference.arXiv preprint arXiv:2310.04378, 2023

    2023 arXiv

  28. [36]

    Mead: A large-scale audio-visual dataset for emotional talking-face generation

    Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. InECCV, Augest 2020

    2020

  29. [37]

    V oxceleb2: Deep speaker recognition.arXiv preprint arXiv:1806.05622, 2018

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. V oxceleb2: Deep speaker recognition.arXiv preprint arXiv:1806.05622, 2018

    2018 arXiv

  30. [38]

    Celebv-hq: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. InEuropean conference on computer vision, pages 650–667. Springer, 2022

    2022

  31. [39]

    Yolov6 v3

    Chuyi Li, Lulu Li, Yifei Geng, Hongliang Jiang, Meng Cheng, Bo Zhang, Zaidan Ke, Xiaoming Xu, and Xiangxiang Chu. Yolov6 v3. 0: A full-scale reloading.arXiv preprint arXiv:2301.05586, 2023

    2023 arXiv

  32. [40]

    Mediapipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019. 12

    1906 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.