Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Real-time One-Step Diffusion-based Expressive Portrait Videos Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that single-step diffusion can generate expressive, lip-synced portrait video at real-time speed with quality close to a 20-step model.

desk verdict A genuinely new one-step distillation recipe for portrait video with plausible numbers, but the evaluation is thin and the main text oversells rolling sampling. read the letter →

arxiv 2412.13479 v1 pith:U2VKCK2A submitted 2024-12-18 cs.CV

Hanzhong Guo , Hongwei Yi , Daquan Zhou , Alexander William Bergman , Michael Lingelbach , Yizhou Yu This is my paper · ORCID
classification cs.CV
keywords one-stepdiffusionsamplinglatentconsistencymodelportraitvideogenerationaudio-driventalkingheadadversarialdistillationeditingfine-tuningreal-timeavatartemporal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces OSA-LCM, a portrait-video diffusion model that generates an expressive talking-head video from one reference image and audio in a single sampling step, instead of the tens of steps current models need. The authors' claim is that one step is enough to match the quality of their own 20-step base model: Table 1 reports FID 23.54 and FVD 124.76 for OSA-LCM at one step against 23.40 and 120.64 for the base at 20 steps. If that holds, portrait-video diffusion becomes real-time, producing one second of video in roughly one second on an H100. The two mechanisms carrying the claim are an adversarial latent consistency model with a portrait-specific discriminator, and a second-stage 'editing fine-tuned' training scheme that reduces the gap between training on noised real frames and inference from pure Gaussian noise.

What carries the argument

The load-bearing object is the adversarial latent consistency model (Adv-LCM), a student network $f_\theta(x_t, t) = c_{\mathrm{skip}}(t) x_t + c_{\mathrm{out}}(t) F_\theta(x_t, t)$ trained so that points on the same probability-flow ODE trajectory map to the same output. In stage one the student is trained with the consistency distillation loss $d(f_\theta(x_t,t), f_{\theta^-}(x^\phi_{t-s}, t-s))$ plus an adversarial loss from a discriminator built from the temporal-attention feature maps of the teacher 3D U-Net, a progressive noise schedule for the discriminator, and a motion loss that matches the frame-difference of generated latents to ground truth. In stage two, EFT turns the student into an editing model: the input is $x^{(n)}_t$, a noised version of the true previous fragment, the target is $x^{(n+1)}_0$, the next fragment, with timesteps sampled only from $[0.8T, T]$; the consistency objective is applied between its prediction and the ODE-solved previous state, along with the same adversarial loss. This editing formulation is what lets the model bridge pure Gaussian noise at one step instead of producing blur.

What would settle it

Compute FVD on a held-out set for one-step generation from pure Gaussian noise versus from noised past frames at a low noise level such as 900; if the Gaussian-start FVD is markedly worse, or motion heatmaps show visibly reduced head movement, the EFT stage has not bridged the temporal gap. A second check is to generate long videos one fragment at a time and measure per-fragment FVD over increasing length; the paper evaluates only isolated clips, so error accumulation would show up as a growing gap.

Watch

Extended reading notes

Core claim

The central discovery is that a latent consistency model can be trained to generate expressive, lip-synced portrait video in exactly one sampling step without losing quality. The authors achieve this by first distilling a 20-step base diffusion model into an adversarial latent consistency model (Adv-LCM) using a consistency loss plus a discriminator that judges temporal quality and audio-lip consistency from features of the teacher 3D U-Net, with a progressive forward process and a motion loss to preserve head movement. In a second stage, they fine-tune the model with the Editing Fine-Tuned method (EFT), which re-frames generation as an editing task: the student receives the noised latent of a true previous video fragment at a high noise level, $t \sim U[0.8T, T]$, and must predict the next fragment, so it learns to change temporal correlation in response to audio and reference-image conditions rather than relying on temporal correlation already present in its input. At inference, starting from pure Gaussian noise, the fine-tuned model transfers this behavior and produces non-blurry one-step video.

Load-bearing premise

The load-bearing premise is that training only on noised true video fragments at high noise levels teaches the model to generate from pure Gaussian noise, where no temporal correlation exists; if that transfer fails, one-step generation reverts to blur and artifacts.

Editorial extensions

If this is right

  • One-step inference makes portrait-video diffusion real-time: the paper reports under one second to generate one second of 512x512 video on an H100, a 20x speedup over its 20-step base model.
  • Quantitative quality at one step is close to the 20-step base on FID (23.54 vs 23.40) and FVD (124.76 vs 120.64), while clearly beating the open-source baselines reported (EchoMimic, Hallo, AniPortrait).
  • The EFT stage also enables autoregressive 'rolling sampling' for long videos, where later fragments start from noised past frames; combined rolling sampling trades identity consistency against temporal naturalness.
  • The approach inherits the base model's ability to handle anime and non-face-detectable inputs, since it does not rely on face detection or masks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If EFT truly closes the temporal gap, the same editing-as-training trick should transfer to other conditional video generation tasks, such as text-to-video or image-to-video, where one-step sampling from Gaussian currently blurs or freezes motion; the authors only demonstrate it for portraits.
  • The combined rolling sampling results suggest a testable trade-off: identity consistency improves as the starting noise level decreases, but FVD worsens; a careful sweep across noise levels and fragment lengths could reveal an optimal operating point for long-video generation.
  • Because the discriminator is built from teacher-model features, the distillation is tied to a specific architecture; a portable version would need to show that a student trained with a feature-based discriminator generalizes across backbones.
  • An open question the paper leaves implicit is how one-step quality degrades over very long generations: EFT is trained on adjacent fragments, and error accumulation across many autoregressive steps could appear even with pure Gaussian starts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces OSA-LCM, a latent consistency model for one-step diffusion-based portrait video generation from a reference image and audio. The method trains an adversarial latent consistency model (Adv-LCM) with consistency, adversarial, and motion losses in Stage 1, then fine-tunes it in Stage 2 using an 'editing fine-tuned' (EFT) scheme where the model maps a noised real previous video fragment to the next fragment. The central claim is that the resulting one-step sampler achieves video quality comparable to a 20-step teacher/base model and to existing open-source portrait video generators, at a 10-20x speedup. The paper includes ablations on the discriminator schedule, motion loss, and sampling-initialization noise levels, and it provides detailed training algorithms.

Significance. If the reported results hold, this is a practically significant contribution: a one-step portrait video diffusion sampler with near-real-time throughput and quality comparable to multi-step diffusion. The paper also proposes a discriminator built on teacher-model features, a progressive forward process, a motion loss, and the EFT fine-tuning scheme. The manuscript is clearly written, includes detailed algorithms that support reproducibility, and provides ablations for several components. However, the significance is tempered by the limited evaluation (100 clips per dataset, no error bars), the omission of a lip-sync metric, and the unverified distribution-shift assumption of EFT; these issues currently make the central claim conditional.

major comments (3)
  1. [Sec. 4.2 / Algorithm 2] The EFT stage trains f_theta only on inputs x(n)_t = alpha_t x(n)_0 + beta_t epsilon with t ~ U[0.8T, T] and a true previous fragment. At inference, OSA-LCM is evaluated on pure Gaussian noise, where no such previous fragment exists. The paper asserts (Sec. 4.2) that EFT 'changes the temporal distribution based on the input of the conditions,' but it provides no quantitative analysis of the residual signal c_skip(t)x(n)_t in Eq. (10), nor any control where the same Adv-LCM is fine-tuned with Gaussian inputs. The text states that a Gaussian-input strategy 'cannot even outperform the default setting, as shown in Tab. 1,' but Table 1 has no such row. Without this control, the ablation comparing OSA-LCM (23.54 FID, 124.76 FVD) to Adv-LCM(+FT) (24.30, 137.24) is the only evidence for EFT; the FVD gap of 12.5 could be within sampling noise, and no significance test is provided. Please add the missing baseline or remove the unsupported claim, and report a quantitative analysis of the temporal-correlation gap (e.g., the effective SNR of the previous-fragment residual at the sampled t range).
  2. [Sec. 5.2 / Table 1] The evaluation uses 100 clips per dataset and reports a single FID/FVD value without error bars or multiple seeds. The key claim that one-step OSA-LCM is comparable to the 20-step base rests on differences of 0.14 FID and 4.12 FVD, which are unlikely to be significant at this sample size; similarly, the improvement of OSA-LCM over Adv-LCM(+FT) is small. Additionally, the paper omits the Sync (lip-sync) metric, justifying this by saying sync is 'excellent among all the methods' (Sec. 5.2). For a portrait video generation system, lip-audio sync is a primary quality attribute and should be measured, especially since distillation can degrade it. Please report confidence intervals or multiple runs, and include a lip-sync metric.
  3. [Sec. 5.2 / Table 1] The comparison with open-source baselines (Hallo, Echomimic, Aniportrait) is not controlled: the baselines use different backbones, resolutions, and training sets, and the paper's base model is retrained specifically on HDTF/VFHQ. The claim that OSA-LCM 'outperforms existing open-source portrait video generation models' is therefore not established. At minimum, the evaluation protocol should be specified for all methods (e.g., same reference frames, same audio, same resolution), and ideally the comparison should include distilling from a shared backbone or controlling for base model quality.
minor comments (5)
  1. [Sec. 6] The sentence 'Our experiments demonstrate the results of using the OSA-LCM with one step can be similar to using the OSA-LCM with one step' appears to contain a typo; likely 'base model with 20 steps' was intended.
  2. [Algorithm 1] The discriminator loss line 'ReLU(1 + x0) + ReLU(1 - ˆx0)' is missing the discriminator and the teacher feature maps; it should be consistent with Eq. (4).
  3. [Eq. (4) and Algorithm 1] The notation for the small noise level is inconsistent: the text says △t ~ U[0,5], Eq. (4) uses alpha_t and beta_t with the main timestep, and Eq. (5) uses alpha_△t. Please clarify the indexing.
  4. [Sec. 1] The phrase 'great process' appears to be a typo for 'great progress.'
  5. [Abstract / Sec. 6] The abstract states 'more than 10x faster' while the conclusion states '20x speedup'; please unify the speedup claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: one-step quality is measured against real videos, and EFT's train/inference gap is an empirical risk rather than a definitional equivalence.

full rationale

The paper's central quantitative claim (Table 1) is anchored externally: FID and FVD are computed against real HDTF/VFHQ clips ('calculate FID and FVD with the original 200 videos'), not against the teacher model's outputs or the student's own training targets. The consistency distillation objective (Eq. 3 and Eq. 9) is a standard self-consistency plus ground-truth regression loss; no reported quantity is defined as a fitted parameter renamed as a prediction. The discriminator is built from the teacher base model's feature maps, but it is trained adversarially on real versus generated latents (Eq. 4), and the final evaluation does not use teacher features as the ground truth; this is a supervision choice, not an equivalence by construction. EFT (Sec. 4.2, Algorithm 2) does train only on noised true previous fragments at t ~ U[0.8T,T], and the paper itself concedes the rolling-sampling variant can underperform and cause ghosting (Supplementary Sec. 7). That is a genuine distribution-shift and generalization risk for the one-step-from-Gaussian claim, but it is an empirical assumption, not a circular derivation: the one-step result is not contained in the EFT training objective by definition. The only self-citation to the authors' PCM [34] is an architectural pointer for the discriminator and is not load-bearing; the method is evaluated against open baselines and real data. No step in the claimed derivation reduces to its own inputs.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

This is a machine-learning methods paper; it introduces new training procedures and model components but no new physical entities, forces, or particles. The 'one-step avatar latent consistency model' and 'editing fine-tuned method' are algorithmic constructs, not entities with external falsifiable handles.

free parameters (7)
  • consistency loss weight gamma = 0.05
    Set in Eq. (9); balances consistency to ground truth.
  • adversarial loss weight lambda = 1.0
    Set in Eq. (9); weighting for discriminator loss.
  • motion loss weight m = 0.07
    Set in Eq. (9); weighting for motion loss.
  • classifier-free guidance scale w = 1.2
    Used in Algorithm 1 and 2 for distillation target.
  • EFT timestep range = [0.8T, T]
    Sec. 4.2 Eq. (10)-(11); restricts stage-2 training to high-noise band.
  • discriminator small noise delta-t = U[0,5]
    Sec. 4.1; noise level added to real and fake data for discriminator input.
  • past frame count = 4
    Sec. 4.1; number of previous frames used as motion condition.
assumptions (4)
  • standard math Consistency model theory: points on the PF-ODE trajectory map to the same x0 and boundary condition f(x0,0)=x0.
    Used in Sec. 2.2, Eq. (1)-(3), to train the Adv-LCM.
  • domain assumption SDEdit-style noising/denoising performs editing: noising the previous fragment and denoising reproduces the next fragment.
    Sec. 4.2 borrows SDEdit [17] to define EFT.
  • domain assumption Wav2Vec audio embeddings are sufficient conditioning for lip-sync and expression.
    Sec. 3 Audio Encoder uses Wav2Vec [25] features.
  • domain assumption The base model (EMO-style) is a high-quality teacher for distillation.
    Sec. 5.1, base model is trained by the authors on HDTF/VFHQ plus anime data; distillation and discriminator depend on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Real-time One-Step Diffusion-based Expressive Portrait Videos Generation." pith.science (2026). https://pith.science/paper/U2VKCK2A

@misc{pith2026241213479,
  author       = {Pith},
  title        = {Pith review of: Real-time One-Step Diffusion-based Expressive Portrait Videos Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2VKCK2A}},
  note         = {Machine review of arXiv:2412.13479}
}
read the original abstract

Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step.

Figures

Figures reproduced from arXiv: 2412.13479 by the authors.

Figure 1
Figure 1. One-step generation results for our proposed one-step avatar latent consistency models (OSA-LCM). We can generate high [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Model architecture for our base model and proposed discriminator. The architecture of the base model refers to the EMO. where θ − is the EMA of function f(·), we take the distance function d as the Huber loss [35] d(xt, x′ t q ) = ∥xt − x ′ t∥ 2 2 + δ 2 −δ, where δ is a threshold hyperparam￾eter. Meanwhile, since the large-scale diffusion models lie in diffusing in the latent space [22], latent consistency mod￾els (… view at source ↗
Figure 3
Figure 3. Different sampling steps for vanilla LCM trained via [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Different sampling steps for Adv-LCM and OSA￾LCM. Adv-LCM will also generate blurry videos. where i denotes the discriminator training iterations, uniform(·) is the uniform distribution. While in the itera￾tions training the latent consistency models, the timesteps tlc…
Figure 5
Figure 5. Figure 5: Two stages training of our OSA-LCM. In the second stage, we propose a novel training scheme that uses the noising data of past frames to predict future frames. Both two stages use adversarial loss and consistency loss. fore, in the case where there is only one sampling…
Figure 6
Figure 6. Figure 6: Motion of heatmap for different sampling steps. It is shown that our Adv-LCM or OSA-LCM can generate a similar motion compared with the base model, and even the motion of OSA-LCM is bigger in some cases. Adv-LCM-4 denotes the heatmap of portrait videos generated by usi…
Figure 7
Figure 7. Figure 7: Comparison between the vanilla sampling and rolling sampling. Left: Vanilla sampling. Right: Rolling sampling. There is a severe ghosting phenomenon, caused by the redundancy of information due to the repeat of the reference image in the first fragment of generated vid…
Figure 9
Figure 9. Figure 9: Comparison between the vanilla sampling and rolling sampling. Left: Vanilla sampling. Right: Rolling sampling. There is a severe ghosting phenomenon, caused by the redundancy of information due to the repeat of the reference image in the first fragment of generated vid…
Figure 10
Figure 10. Figure 10: Comparison between the different sampling steps. The results of the generated portrait video via OSA-LCM with one sampling step show similar quality and motion expressivity with the generated videos via the base model with 20 sampling steps, which means a speed-up of …
Figure 11
Figure 11. Figure 11: Comparison among different models with only one sampling step. OSA-LCM can generate high-quality of portrait videos with one sampling step, while other models will generate different types of artifacts with limited sampling steps. 4 [PITH_FULL_IMAGE:figures/full_fig_…
Figure 12
Figure 12. Figure 12: Comparison with different methods of generated portrait videos based on diffusion models. OSA-LCM can outperform other portrait video diffusion models while using only one step of sampling and shows strong robustness over different audio types and reference images, de…
Figure 13
Figure 13. Figure 13: More results of OSA-LCM with one sampling step. 6 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decomposing Subject-Driven Image Generation via Intermediate Structural Prediction

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    A two-stage method predicts an intermediate Canny map for structure then renders the image conditioned on appearance and structure, paired with a 100k text-aware dataset, to improve detail preservation in subject-driv...

Reference graph

Works this paper leans on

47 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Analytic- dpm: an analytic estimate of the optimal reverse vari- ance in diffusion probabilistic models

    Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic- dpm: an analytic estimate of the optimal reverse vari- ance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503, 2022. 2

    arXiv 2022

  2. [2]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, et al. Video generation models as world simulators, 2024. 1

    work page 2024

  3. [3]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 4

    2021

  4. [4]

    Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024. 1, 2, 3, 8

    arXiv 2024

  5. [5]

    Out of time: auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: auto- mated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017. 7

    work page 2016

  6. [6]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learn- ing, 2024. 1

    2024

  7. [7]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 4

    work page 2020

  8. [8]

    Improved training of wasserstein gans

    Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017. 4

    work page 2017

Show all 47 references
  1. [9]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 3, 5

    2023 arXiv

  2. [10]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 7

    2017

  3. [11]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  4. [12]

    Videobooth: Diffusion-based video generation with image prompts

    Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6689– 6700, 2024. 1

    2024

  5. [13]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 2

    2022

  6. [14]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775–5787,

  7. [15]

    Latent consistency models: Synthesizing high- resolution images with few-step inference

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023. 2, 3

    2023 arXiv

  8. [16]

    Osv: One step is enough for high-quality image to video generation

    Xiaofeng Mao, Zhengkai Jiang, Fu-Yun Wang, Wenbing Zhu, Jiangning Zhang, Hao Chen, Mingmin Chi, and Yabiao Wang. Osv: One step is enough for high-quality image to video generation. arXiv preprint arXiv:2409.11367 , 2024. 2, 4

    2024 arXiv

  9. [17]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. arXiv preprint arXiv:2108.01073, 2021. 6

    2021 arXiv

  10. [18]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14297–14306, 2023. 2

    2023

  11. [19]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  12. [20]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia , pages 484–492, 2020. 3

    2020

  13. [21]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 1

    2021

  14. [22]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 3

    2022

  15. [23]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  16. [24]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  17. [25]

    wav2vec: Unsupervised pre-training for speech recognition

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019. 3

    1904 arXiv

  18. [26]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2

    2015

  19. [27]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2

    2010 arXiv

  20. [28]

    Generative modeling by esti- mating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 2

    2019

  21. [29]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. arXiv preprint arXiv:2011.13456, 2020. 2

    2011 arXiv

  22. [30]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 2

    2023 arXiv

  23. [31]

    Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior

    Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841, 2023. 1, 2, 3

    2023 arXiv

  24. [32]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 1, 2, 3, 5, 7

    2024 arXiv

  25. [33]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 7

    2019

  26. [34]

    Phased consistency model

    Fu-Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingel- bach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency model. arXiv preprint arXiv:2405.18407, 2024. 4

    2024 arXiv

  27. [35]

    Animatelcm: Accelerating the animation of personalized diffusion mod- els and adapters with decoupled consistency learning

    Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Accelerating the animation of personalized diffusion mod- els and adapters with decoupled consistency learning. arXiv preprint arXiv:2402.00769, 2024. 3

    2024 arXiv

  28. [36]

    Videolcm: Video latent consistency model

    Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolcm: Video latent consistency model. arXiv preprint arXiv:2312.09109,

    arXiv

  29. [37]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024. 1, 2, 3, 8

    2024 arXiv

  30. [38]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 1, 2, 3, 8

    2024 arXiv

  31. [39]

    Vasa-1: Lifelike audio-driven talking faces generated in real time

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404.10667 ,

    arXiv

  32. [40]

    Motion consistency model: Accel- erating video diffusion with disentangled motion-appearance distillation

    Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li, Jian- feng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan, and Lijuan Wang. Motion consistency model: Accel- erating video diffusion with disentangled motion-appearance distillation. arXiv preprint arXiv:2406.06890, 2024. 6, 7

    2024 arXiv

  33. [41]

    Metaportrait: Identity-preserving talking head gener- ation with fast personalized adaptation

    Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, Hsiang- Tao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: Identity-preserving talking head gener- ation with fast personalized adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2023

  34. [42]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 1

    2023

  35. [43]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  36. [44]

    Denoising diffusion bridge models

    Linqi Zhou, Aaron Lou, Samar Khanna, and Stefano Er- mon. Denoising diffusion bridge models. arXiv preprint arXiv:2309.16948, 2023. 1 10 Real-time One-Step Diffusion-based Expressive Portrait Videos Generation Supplementary Material

    2023 arXiv

  37. [45]

    However, in the experiment, we found that rolling sampling for the Adv-LCM performed worse than the base Adv-LCM in one sampling step

    More discussion about the rolling sampling We have discussed one of the advantages of our proposed EFT: It can use rolling sampling, which adds noise to the past n frames, and utilize the noisy data as the start of sam- pling instead of the Gaussian, which is one kind of auto-...

  38. [46]

    10 and Fig

    More comparison between different sam- pling steps For the sake of experimental completeness, we show in Fig. 10 and Fig. 11 the generation results of our method under unsynchronized counts, specifically using our base model with the DDIM scheduler [27], the Adv-LCM with LCM s...

  39. [47]

    In addition, in this sec- tion in order to show the robustness of different approaches to audio types, we will choose to use songs as the audio to drive

    More comparison between different meth- ods Furthermore, for the sake of completeness, we also carry out a comparison of our OSA-LCM with other methods, specifically, we utilized Hallo to reason 40 steps using the DDIM scheduler, and Echomimic’s accelerated version (Echo-Acc) ...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.