Pith. sign in

REVIEW 3 major objections 4 minor 5 cited by

StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read StableAvatar generates infinite-length, audio-synchronized avatar videos end-to-end, preserving reference identity beyond 3500 frames.

desk verdict The audio adapter idea is real and the ablations back it, but the 'infinite-length without post-processing' claim collapses on contact with the method's own sliding-window fusion. read the letter →

arxiv 2508.08248 v1 pith:AIU6S2CD submitted 2025-08-11 cs.CV

Shuyuan Tu , Yueming Pan , Yinming Huang , Xintong Han , Zhen Xing , Qi Dai , Chong Luo , Zuxuan Wu
show 1 more author
Yu-Gang Jiang
This is my paper · ORCID
classification cs.CV
keywords audio-drivenavatarvideogenerationdiffusiontransformerinfinite-lengthlatentdistributionerroraccumulationtimestep-awareaudioadapternativeguidancelipsynchronizationidentityconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StableAvatar sets out to remove the roughly 15-second ceiling on audio-driven avatar video generation. The paper's diagnosis is that existing models degrade on long videos because they inject off-the-shelf audio embeddings directly into a diffusion backbone that has no audio priors, so every generated segment leaves the latent distribution slightly off target and the error accumulates across segments until faces distort, colors drift, and identity washes out. Its proposed cure is a Timestep-aware Audio Adapter that conditions the audio representation on the denoising state itself, making the model learn a joint audio-latent distribution at each step, supported by an inference-time Audio Native Guidance that sharpens lip-sync and a dynamic weighted sliding-window fusion that smooths clip boundaries. If the claim holds, avatar videos of minutes to hours can be generated in a single pass from one reference image and an audio track, with identity and lip-sync preserved past 3500 frames, using a 1.3-billion-parameter model rather than a 14-billion one. The supporting experiments report large margins over the strongest competitor on the authors' private Long100 benchmark (identity similarity 0.849 versus 0.471).

What carries the argument

The load-bearing object is the Timestep-aware Audio Adapter. Raw Wav2Vec embeddings are concatenated with neighboring audio context frames, projected by an MLP, and then used as Query in a cross-attention block whose Key and Value are the noisy latents $z_t$; the outputs are scaled and shifted by the same timestep embeddings the DiT uses internally. Because diffusion pretraining ties each timestep embedding to a specific latent distribution, this modulation implicitly binds the audio representation to the latent distribution of the current denoising step, producing refined embeddings $\bar{a}_t$ that are then injected into the backbone by cross-attention. The inference mechanisms build on the same idea: Audio Native Guidance rewrites the denoising score as a weighted combination of the joint audio-latent prediction, the unconditional prediction, and the audio-conditioned latent prediction, treating $\bar{a}_t$ as an extra model output; the Dynamic Weighted Sliding-Window strategy blends overlapping latent windows with logarithmically spaced weights so both edges of every window are fused from both sides. The adapter is what carries the argument: it converts audio conditioning from an external signal the backbone distrusts into a function of the model's own evolving latent state.

What would settle it

Generate one audio-driven video of 3500+ frames with StableAvatar and one with the same backbone but direct audio injection, then compare the denoised-latent distributions of frames 1-200 and frames 3500-3700 with FID or a divergence measure: the paper's error-accumulation story predicts near-stationarity for StableAvatar and clear drift for the baseline. A public head-to-head using released competitor checkpoints on a public long-video set, without the private retraining protocol, would separately settle whether the reported margins survive.

Watch

Extended reading notes

Core claim

The central claim, put as the authors would put it, is that the long-video failure of audio-driven avatar models is not a defect of the video backbone but of the audio conditioning path. Since diffusion backbones carry no audio priors, embeddings from an off-the-shelf extractor that are injected by cross-attention push each successive video segment's latents slightly off the target distribution; across many segments the error accumulates, and identity, color, and lip-sync collapse. StableAvatar's answer is to make the audio conditioning depend on the denoising state: the Timestep-aware Audio Adapter lets audio tokens cross-attend with the noisy latents and modulates them with the timestep embeddings, so the conditioning signal is forged from the joint audio-latent distribution at every step rather than handed in from outside. At inference, Audio Native Guidance replaces classifier-free guidance by steering the sampling score toward that same joint distribution with the refined audio embedding treated as an additional prediction, and a dynamic weighted sliding-window strategy fuses overlapping latent windows to keep long videos smooth. On Long100, the paper's benchmark of 100 unseen videos of two to five minutes, the 1.3B-parameter StableAvatar is reported to keep identity similarity at 0.849 and lip-sync at 8.24 while the strongest Wan2.1-14B-based competitor, OmniAvatar, drops to 0.471 and 4.45; the model is reported stable past 3500 frames in a single pass.

Load-bearing premise

The headline margins rest on the assumption that retraining every competitor on the authors' private 1200-hour dataset and judging them on the private Long100 benchmark is a genuinely fair comparison, rather than a protocol that happens to favor StableAvatar's design.

Editorial extensions

If this is right

  • Avatar videos longer than 15 seconds no longer require clip stitching or post-processing: the authors report quality that stays flat across thousands of frames, making video length a matter of compute and memory rather than drift.
  • A 1.3B-parameter model can beat 14B-based systems on long-form avatar video, because the binding constraint is the conditioning path, not scale.
  • Inference-time smoothing alone is not enough: the ablations show motion-frame and conventional sliding-window strategies still collapse to FVD values near 1800-2000 and CSIM near 0.4, whereas the full model reaches 504 and 0.849.
  • Audio Native Guidance outperforms classifier-free guidance at no training cost (Sync-C 8.20 versus 7.62) and skips image and text as guidance inputs, saving memory.
  • The same recipe claims roughly half the GPU memory and about ten times the inference speed of the leading competitor, because the effective fix is architectural, not a bigger backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal story is right, the recipe generalizes: any diffusion model that injects off-the-shelf embeddings from a second modality into latent space without cross-modal priors should show the same cumulative drift in long-form generation, and a timestep-aware adapter is a portable cure for text-, pose-, or emotion-conditioned video.
  • The paper's own 'w/Random modulation' ablation (CSIM 0.632 versus 0.849) isolates the timestep coupling as the source of most of the gain; re-running that ablation across datasets and seeds would confirm it is the coupling, not the extra parameters, doing the work.
  • The paper itself reports a boundary case: given a reference far outside human morphology, such as a fantastical creature, the model cannot locate the lips and lip-sync fails; the infinite-length guarantee is therefore implicitly scoped to human-like references and would need an explicit reference-semantics module to extend further.
  • A decisive independent check would re-run the comparison with released competitor checkpoints on a public long-video benchmark, since the reported margins come from retraining all models on the authors' private 1200-hour dataset and scoring on the private Long100.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. StableAvatar proposes an audio-driven avatar video generation system built on the Wan2.1-1.3B video diffusion transformer, with three components: a Timestep-aware Audio Adapter that refines Wav2Vec embeddings through timestep-conditioned modulation and cross-attention with latents; an Audio Native Guidance mechanism intended to replace classifier-free guidance using a joint audio-latent score; and a Dynamic Weighted Sliding-Window Strategy (DWSW) that fuses overlapping window latents during inference. The paper reports state-of-the-art results on HDTF, AVSpeech, and a privately collected Long100 benchmark, with quantitative gains in identity consistency (CSIM) and lip synchronization (Sync-C) over competitors including Wan2.1-14B-based OmniAvatar. The authors claim that StableAvatar is the first end-to-end diffusion transformer capable of infinite-length, post-processing-free video generation, supported by qualitative demos of videos exceeding 3500 frames.

Significance. If the empirical claims hold, this is a practically significant contribution: it would demonstrate that a 1.3B-parameter video diffusion model can maintain identity and audio-lip synchronization for multi-minute avatar videos, at substantially lower inference cost than 14B-parameter competitors. The ablations are informative and support the qualitative claim that each proposed component degrades performance when removed, and the supplementary limitation section (Sec. A.10) and failure-case figure are honest disclosures. However, the headline claims of 'infinite-length' and 'without post-processing' are not supported by the algorithm as presented, and the SOTA comparison rests on an unverifiable retraining protocol on a private dataset. The paper is a solid engineering contribution in need of claim revision and additional validation before it can be accepted.

major comments (3)
  1. [Abstract; Sec. 3.3 and Algorithm 1] The claim 'infinite-length high-quality videos without post-processing' is contradicted by the paper's own inference procedure. Algorithm 1 takes a fixed total latent length L as input, denoises overlapping windows, and at each timestep fuses latents from adjacent windows with a logarithmic weighting and injects the fused latents back into both windows. This is an explicit inference-time stitching/fusion step performed in latent space. Table 6 shows the step is not cosmetic: removing DWSW raises FVD from 532 to 718. The central claim should be revised to 'finite-length long-video generation with a latent sliding-window fusion step'; no streaming or autoregressive extension protocol is given, so 'infinite-length' is not supported by the algorithm or by the 3-5 minute demos.
  2. [Sec. 3.2, Eqs. (6)-(10)] The Audio Native Guidance derivation contains unstated approximations and notational inconsistencies. Eq. (6) factorizes the modified sampling density into pθ(A|[zt,ā])^α pθ(ā|zt,A)^β without stating the assumed conditional-independence structure. In going from Eq. (7) to Eq. (9), the score function mixes three different conditioning sets: ∇log pθ([zt,ā]|A), ∇log pθ([zt,ā]), and ∇log pθ(zt|A). The final form in Eq. (10), however, implements the last term as D([zt,∅], y, I, A;θ), whose conditioning includes text and image inputs and whose first argument [zt,∅] does not match pθ(zt|A). The authors should state precisely which density each term denotes and justify the approximation that ∇log pθ(zt|A) can be evaluated by the network with the audio as the only conditioning signal.
  3. [Sec. 4.1-4.2 and Table 1] The quantitative superiority claim rests on a comparison protocol that cannot be independently verified. All baselines are retrained on the authors' private 1200-hour dataset and evaluated on the private Long100 benchmark, but no details are given about training budgets, hyperparameters, checkpoints, or the selection and prompt protocol for Long100. The reported margins on Long100 (CSIM 0.849 vs. 0.471; Sync-C 8.24 vs. 4.45) could be inflated by undertuned baselines or by benchmark choices that favor the proposed design. Additionally, no error bars or statistical significance tests are provided for any table. The authors should release the Long100 protocol or report multiple seeds and variance, and should provide per-baseline training details.
minor comments (4)
  1. [Tables 2, 5, 6] The same 'Ours' configuration on Long100 reports FVD=504 in Table 2 but FVD=532 in Tables 5 and 6; the discrepancy is not explained and should be clarified (e.g., different seeds, evaluation subsets, or corrected numbers).
  2. [Sec. 3.3] The text says 'previous sling-window denoising strategy'; 'sling' should be 'sliding'. Also, the pseudo-code in Algorithm 1 uses 'np.linspace(0, 1, num samples=m)' and a subsequent normalization that is formatted ambiguously; the notation should be cleaned up.
  3. [Sec. 1, contributions] The contributions list skips item (3): items are numbered (1), (2), (4). This is a minor editorial issue.
  4. [Table 3] The row label 'w/o Aduio Adapter' contains a typo and should read 'w/o Audio Adapter'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: all modules are independently ablated and benchmarked; the 'without post-processing' phrasing conflicts with DWSW but is a claim-consistency issue, not a circularity.

full rationale

The claimed derivation chain is not circular. The Audio Adapter (Sec. 3.1) is a trained module with an ablation (Table 3) showing degradation without it; the guidance (Sec. 3.2) is a heuristic score modification with hyperparameters alpha and beta and a comparison to CFG (Table 5); and DWSW (Sec. 3.3) is an inference-time fusion whose contribution is verified by Table 6. None of these quantities are defined in terms of the target metrics, and no benchmark numbers are fitted and then renamed as predictions. The SOTA comparison retrains all competitors on the authors' private dataset and evaluates on the private Long100 benchmark, which is a fairness concern rather than a circularity concern. The paper's own stated limitation (Sec. A.10, failure on non-human creatures) is honest and does not smuggle in a result. The only notable overclaim is 'without post-processing' in the Abstract versus the overlapping-latent fusion of Algorithm 1; that is a semantic contradiction about what counts as post-processing, not a derivation that reduces to its input. No load-bearing self-citation or imported uniqueness theorem appears, so the central empirical claims remain independently testable.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical modules rather than a formal derivation. Free parameters are hand-selected hyperparameters (guidance scales, context length, window sizes) whose values are partially undisclosed. Key domain assumptions about audio-prior scarcity and timestep-latent correlation are asserted rather than measured. No invented entities are introduced.

free parameters (3)
  • Audio Native Guidance scales alpha, beta = alpha=4.5, beta=3.0
    Guidance strengths chosen by hand (Sec 4.1); no sensitivity analysis or fitting procedure is reported.
  • Audio context length 2k+1 = unspecified
    The number of neighboring audio frames concatenated for each timestep (Eq 1) is a hyperparameter; its value is not stated.
  • DWSW window length l and overlap m = unspecified
    Sliding-window fusion parameters in Algorithm 1 whose concrete values are not given.
assumptions (4)
  • domain assumption Timestep embeddings are strongly correlated with latent distributions.
    Used in Sec 3.1 to justify timestep-aware modulation; asserted without verification.
  • domain assumption Current diffusion backbones lack audio-related priors.
    Central motivation in Sec 1 and 3.1; not directly measured.
  • standard math Standard Rectified Flow diffusion forward and reverse processes (Eq 12-13 in Supp).
    Background framework borrowed from prior work.
  • ad hoc to paper The Bayesian factorization in Eq 6 assumes conditional independence and that the audio-latent estimate can be treated as an additional prediction target.
    Underlies Audio Native Guidance; the required conditionals are not shown to be computable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation." pith.science (2026). https://pith.science/paper/AIU6S2CD

@misc{pith2026250808248,
  author       = {Pith},
  title        = {Pith review of: StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AIU6S2CD}},
  note         = {Machine review of arXiv:2508.08248}
}
read the original abstract

Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency. This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing. Conditioned on a reference image and audio, StableAvatar integrates tailored training and inference modules to enable infinite-length video generation. We observe that the main reason preventing existing models from generating long videos lies in their audio modeling. They typically rely on third-party off-the-shelf extractors to obtain audio embeddings, which are then directly injected into the diffusion model via cross-attention. Since current diffusion backbones lack any audio-related priors, this approach causes severe latent distribution error accumulation across video clips, leading the latent distribution of subsequent segments to drift away from the optimal distribution gradually. To address this, StableAvatar introduces a novel Time-step-aware Audio Adapter that prevents error accumulation via time-step-aware modulation. During inference, we propose a novel Audio Native Guidance Mechanism to further enhance the audio synchronization by leveraging the diffusion's own evolving joint audio-latent prediction as a dynamic guidance signal. To enhance the smoothness of the infinite-length videos, we introduce a Dynamic Weighted Sliding-window Strategy that fuses latent over time. Experiments on benchmarks show the effectiveness of StableAvatar both qualitatively and quantitatively.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    LeapTalk distills a multi-step diffusion teacher into a one-step Brownian-bridge student and reports stable streaming talking-head generation at up to 200 FPS.

  2. Vidu S1: A Real-Time Interactive Video Generation Model

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Vidu S1 generates voice-controlled interactive avatar video in real time at 540p/42 FPS with claimed infinite stable streams and top reported quality metrics.

  3. Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Live Avatar reports real-time streamable generation from a 14B audio-driven diffusion model at ~20 FPS on 5 H800s with stable identity over 10,000 seconds.

  4. TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Anchor-guided fixed-capacity multimodal memory plus stage-parallel few-step denoising stabilizes long-form causal audio-video digital humans at real-time speed.

  5. InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Sparse-frame dubbing with adjacent-chunk keyframe sampling lets a streaming audio-video model produce full-body motion synchronized to new audio while preserving identity and camera motion.

Reference graph

Works this paper leans on

84 extracted references · 28 canonical work pages · cited by 5 Pith papers

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 2020. 2, 3, 4

    2020

  2. [2]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024. 2

    arXiv 2024

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  4. [4]

    Generating long videos of dynamic scenes

    Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. In NIPS, 2022. 3

    2022

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  6. [6]

    Hunyuanvideo-avatar: High-fidelity audio-driven hu- man animation for multiple characters

    Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, and Qinglin Lu. Hunyuanvideo-avatar: High-fidelity audio-driven hu- man animation for multiple characters. arXiv preprint arXiv:2505.20156, 2025. 1, 3, 5, 6, 8, 2

    arXiv 2025

  7. [7]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. In AAAI,

  8. [8]

    Videoretalking: Audio-based lip synchronization for talking head video editing in the wild

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia,

    work page

Show all 84 references
  1. [9]

    Out of time: auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: auto- mated lip sync in the wild. In ACCV, 2016. 5, 1

    2016

  2. [10]

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. In CVPR, 2025. 1, 2, 3, 5, 6, 8

    2025

  3. [11]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019. 1

    2019

  4. [12]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021. 1, 2

    2021

  5. [13]

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech sepa- ration

    Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech sepa- ration. arXiv preprint arXiv:1804.03619, 2018. 5

    2018 arXiv

  6. [14]

    Omniavatar: Efficient audio-driven avatar video generation with adaptive body animation

    Qijun Gan, Ruizi Yang, Jianke Zhu, Shaofei Xue, and Steven Hoi. Omniavatar: Efficient audio-driven avatar video generation with adaptive body animation. arXiv preprint arXiv:2506.18866, 2025. 1, 2, 3, 5, 6, 8

    2025 arXiv

  7. [15]

    Toontalker: Cross-domain face reenactment

    Yuan Gong, Yong Zhang, Xiaodong Cun, Fei Yin, Yanbo Fan, Xuan Wang, Baoyuan Wu, and Yujiu Yang. Toontalker: Cross-domain face reenactment. In ICCV, 2023. 2

    2023

  8. [16]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 2020. 2

    2020

  9. [17]

    Stylesync: High-fidelity generalized and personalized lip sync in style-based genera- tor

    Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based genera- tor. In CVPR, 2023. 2

    2023

  10. [18]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning. In ICLR,

  11. [19]

    Flexible diffusion modeling of long videos

    William Harvey, Saeid Naderiparizi, Vaden Masrani, Chris- tian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In NIPS, 2022. 3

    2022

  12. [20]

    Streamingt2v: Con- sistent, dynamic, and extendable long video generation from text

    Roberto Henschel, Levon Khachatryan, Hayk Poghosyan, Daniil Hayrapetyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Con- sistent, dynamic, and extendable long video generation from text. In CVPR, 2025. 3

    2025

  13. [21]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 1, 2

    2022 arXiv

  14. [22]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. NIPS, 2017. 5

    2017

  15. [23]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 2, 3, 4, 8

    2022 arXiv

  16. [24]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 1, 2

    2020

  17. [25]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 2022. 1, 2

    2022

  18. [26]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 2

    2022 arXiv

  19. [27]

    Sonic: Shifting focus to global audio perception in portrait animation

    Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chum- ing Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonic: Shifting focus to global audio perception in portrait animation. In CVPR, 2025. 1, 2, 3, 5, 6, 8 9

    2025

  20. [28]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024. 2

    2024 arXiv

  21. [29]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 2, 3

    2024 arXiv

  22. [30]

    Let them talk: Audio-driven multi-person conversa- tional video generation

    Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xi- aoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conversa- tional video generation. arXiv preprint arXiv:2505.22647 ,

    arXiv

  23. [31]

    Latentsync: Audio conditioned latent diffusion models for lip sync

    Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. Latentsync: Audio conditioned latent diffusion models for lip sync. arXiv e- prints, pages arXiv–2412, 2024. 2

    2024

  24. [32]

    Magicmotion: Controllable video genera- tion with dense-to-sparse trajectory guidance

    Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, and Zuxuan Wu. Magicmotion: Controllable video genera- tion with dense-to-sparse trajectory guidance. arXiv preprint arXiv:2503.16421, 2025. 1

    2025

  25. [33]

    Cyberhost: A one-stage diffusion framework for audio-driven talking body generation

    Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Ji- aqi Yang, Zerong Zheng, and Yanbo Zheng. Cyberhost: A one-stage diffusion framework for audio-driven talking body generation. In ICLR, 2025. 3

    2025

  26. [34]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.01061, 2025. 3

    2025 arXiv

  27. [35]

    Flow matching for generative mod- eling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. arXiv preprint arXiv:2210.02747, 2022. 1

    2022 arXiv

  28. [36]

    Phantom: Subject-consistent video generation via cross- modal alignment

    Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Ji- awei Liu, Gen Li, Siyu Zhou, Qian He, and Xinglong Wu. Phantom: Subject-consistent video generation via cross- modal alignment. arXiv preprint arXiv:2502.11079 , 2025. 5

    2025 arXiv

  29. [37]

    Freelong++: Training-free long video generation via multi-band spectralfusion

    Yu Lu and Yi Yang. Freelong++: Training-free long video generation via multi-band spectralfusion. arXiv preprint arXiv:2507.00162, 2025. 3

    2025 arXiv

  30. [38]

    Free- long: Training-free long video generation with spectralblend temporal attention

    Yu Lu, Yuanzhi Liang, Linchao Zhu, and Yi Yang. Free- long: Training-free long video generation with spectralblend temporal attention. In NIPS, 2024. 3

    2024

  31. [39]

    Medi- apipe: A framework for building perception pipelines

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Medi- apipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019. 5

    1906 arXiv

  32. [40]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. In ICLR, 2021. 1, 2

    2021

  33. [41]

    Echomimicv2: Towards striking, simplified, and semi-body human animation

    Rang Meng, Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimicv2: Towards striking, simplified, and semi-body human animation. In CVPR, 2025. 1, 3

    2025

  34. [42]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021. 2

    2021

  35. [43]

    Dpe: Disen- tanglement of pose and expression for general video portrait editing

    Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xi- aodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Disen- tanglement of pose and expression for general video portrait editing. In CVPR, 2023. 2

    2023

  36. [44]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 2

    2023

  37. [45]

    Freenoise: Tuning-free longer video diffusion via noise rescheduling.arXiv preprint arXiv:2310.15169, 2023

    Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xin- tao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling.arXiv preprint arXiv:2310.15169, 2023. 3

    2023 arXiv

  38. [46]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 3

    2021

  39. [47]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 1, 2

    2022

  40. [48]

    The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations

    Gaurav Sharma, Wencheng Wu, and Edul N Dalal. The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian...

    2005

  41. [49]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  42. [50]

    Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elho- seiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In CVPR, 2022. 3

    2022

  43. [51]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In ICLR, 2021. 1, 2

    2021

  44. [52]

    Linsen Song, Wayne Wu, Chaoyou Fu, Chen Change Loy, and Ran He. Audio-driven dubbing for user generated con- tents via style-aware semi-parametric synthesis.IEEE Trans- actions on Circuits and Systems for Video Technology, 33(3): 1247–1261, 2022. 2

    2022

  45. [53]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. In ICLR, 2021. 1, 2, 4

    2021

  46. [54]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In ECCV, 2024. 2, 4

    2024

  47. [55]

    Emo2: End-effector guided audio-driven avatar video generation

    Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio-driven avatar video generation. arXiv preprint arXiv:2501.10687, 2025. 3

    2025 arXiv

  48. [56]

    Nonlinear 3d face morphable model

    Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. In CVPR, 2018. 2 10

    2018

  49. [57]

    Consistent view synthesis with pose-guided diffusion models

    Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia- Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. In CVPR, 2023. 3

    2023

  50. [58]

    Implicit temporal modeling with learn- able alignment for video recognition

    Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu-Gang Jiang. Implicit temporal modeling with learn- able alignment for video recognition. In ICCV, 2023. 2

    2023

  51. [59]

    Motioneditor: Editing video motion via content-aware diffusion

    Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motioneditor: Editing video motion via content-aware diffusion. In CVPR, 2024. 1, 2

    2024

  52. [60]

    Motionfollower: Editing video motion via lightweight score-guided diffusion

    Shuyuan Tu, Qi Dai, Zihao Zhang, Sicheng Xie, Zhi-Qi Cheng, Chong Luo, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motionfollower: Editing video motion via lightweight score-guided diffusion. arXiv preprint arXiv:2405.20325 , 2024

    2024 arXiv

  53. [61]

    Stableanimator: High- quality identity-preserving human image animation

    Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, and Zuxuan Wu. Stableanimator: High- quality identity-preserving human image animation. In CVPR, 2025. 2

    2025

  54. [62]

    Sta- bleanimator++: Overcoming pose misalignment and face distortion for human image animation

    Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Sta- bleanimator++: Overcoming pose misalignment and face distortion for human image animation. arXiv preprint arXiv:2507.15064, 2025

    2025 arXiv

  55. [63]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023. 1, 2

    2023

  56. [64]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5

    2018 arXiv

  57. [65]

    Mcvd-masked conditional video diffusion for prediction, generation, and interpolation

    Vikram V oleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. In NIPS, 2022. 3

    2022

  58. [66]

    Wan: Open and advanced large-scale video generative models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, P...

    2025 arXiv

  59. [67]

    V-express: Conditional dropout for progres- sive training of portrait video generation

    Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progres- sive training of portrait video generation. arXiv preprint arXiv:2406.02511, 2024. 1, 2

    2024 arXiv

  60. [68]

    Gen-l-video: Multi-text to long video generation via temporal co-denoising

    Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305.18264, 2023. 3

    2023 arXiv

  61. [69]

    Fanta- sytalking: Realistic talking portrait generation via coherent motion synthesis

    Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yun- peng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fanta- sytalking: Realistic talking portrait generation via coherent motion synthesis. arXiv preprint arXiv:2504.04842 , 2025. 1, 2, 3, 5, 6, 8

    2025 arXiv

  62. [70]

    Magicvideo-v2: Multi- stage high-aesthetic video generation

    Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magicvideo-v2: Multi- stage high-aesthetic video generation. arXiv preprint arXiv:2401.04468, 2024. 2

    2024 arXiv

  63. [71]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024. 1, 2, 5, 6

    2024 arXiv

  64. [72]

    Genrec: Unifying video generation and recognition with diffusion models

    Zejia Weng, Xitong Yang, Zhen Xing, Zuxuan Wu, and Yu-Gang Jiang. Genrec: Unifying video generation and recognition with diffusion models. arXiv preprint arXiv:2408.15241, 2024. 1

    2024 arXiv

  65. [73]

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching lmms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17090, 2023. 5, 1

    2023 arXiv

  66. [74]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In CVPR, 2023. 2

    2023

  67. [75]

    Simda: Simple diffusion adapter for efficient video generation

    Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7827– 7839, 2024. 1

    2024

  68. [76]

    Aid: Adapting image2video diffusion mod- els for instruction-guided video prediction

    Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, and Yu- Gang Jiang. Aid: Adapting image2video diffusion mod- els for instruction-guided video prediction. arXiv preprint arXiv:2406.06465, 2024

    2024 arXiv

  69. [77]

    A survey on video dif- fusion models

    Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video dif- fusion models. ACM Computing Surveys, 57(2):1–42, 2024. 1

    2024

  70. [78]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 2, 3

    2024 arXiv

  71. [79]

    Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan

    Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In ECCV,

  72. [80]

    Nuwa-xl: Diffusion over diffusion for extremely long video generation.arXiv preprint arXiv:2303.12346, 2023

    Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation.arXiv preprint arXiv:2303.12346, 2023. 3 11

    2023 arXiv

  73. [81]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In CVPR, 2023. 1, 2, 5, 6

    2023

  74. [82]

    Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset. In CVPR, 2021. 5

    2021

  75. [83]

    CelebV- HQ: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV- HQ: A large-scale video facial attributes dataset. In ECCV,

  76. [2022]

    The pipeline of our DWSW

    5, 1 12 ������������������ ���������� � ������������������ �������������� � ���� ������������������������� ��� ���� ���� ���� ���� ���� ���� ���� ���� ��� ��� ���� ���� ���� ���� ���� ���� ���� ���� ��� ������������������������� ����������������������������������� ���������� �...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.