Pith. sign in

REVIEW 5 major objections 5 minor 8 cited by

Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MultiTalk binds each audio track to its speaker in two-person conversational videos using matching rotary-position labels.

desk verdict The task is new and the mechanism is plausible, but the central binding claim rests on a static left/right evaluation split that undercuts the adaptive-localization premise. read the letter →

arxiv 2505.22647 v1 pith:GHA2MR6O submitted 2025-05-28 cs.CV

Zhe Kong , Feng Gao , Yong Zhang , Zhuoliang Kang , Xiaoming Wei , Xunliang Cai , Guanying Chen , Wenhan Luo This is my paper · ORCID
classification cs.CV
keywords multi-personconversationalvideogenerationaudio-drivenhumananimationLabelRotaryPositionEmbeddingaudio-personbindingdiffusiontransformerinstructionfollowinglipsynchronizationimage-to-video
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a new task: generating a video of two people holding a conversation, driven by two separate audio tracks. The authors argue that the core obstacle is audio–person binding: earlier models make all faces speak at once from a single track, and a naive left/right split fails once a person moves. Their framework, MultiTalk, solves binding with Label Rotary Position Embedding (L-RoPE), a labelling scheme that makes each audio stream attend only to the tokens of its designated person. They also report that training only the newly added audio layers while mixing in image-to-video data preserves the base model's ability to follow instructions such as 'the woman takes a sip.' Working multi-person audio-driven video would matter for multi-character movie scenes and livestreaming.

What carries the argument

The load-bearing object is Label Rotary Position Embedding (L-RoPE), an extension of rotary position embedding in which each token's rotation is driven by a semantic label rather than a spatial coordinate. In the audio cross-attention layer, queries come from video latents and keys/values from the two audio streams; each token's query is rotated by $\theta_i = l_i \theta_{\mathrm{base}}$, with labels chosen so that a person's video tokens and that person's audio share nearby labels. Behind this sits Adaptive Person Localization, which uses the reference-image-to-video self-attention map of the diffusion transformer to compute per-token similarity to each subject mask, giving each video token a person label. That label, in turn, is what L-RoPE needs to rotate the query correctly.

What would settle it

Generate a two-person clip where the two people walk toward each other, swap positions, and continue talking, then check whether each person's lips keep following their original audio stream; if the lip-sync switches speakers as the people cross, the localization assumption is violated.

Watch

Extended reading notes

Core claim

The central claim is that multi-stream audio can be correctly bound to the right person by giving both audio tokens and video-latent tokens a label, then rotating the video queries by an angle $\theta_i = l_i \theta_{\mathrm{base}}$ in the audio cross-attention layer. The video token labels are derived adaptively from the reference-image-to-video self-attention map: each token is assigned to the person whose mask has the highest average similarity, and its label is a normalized similarity value inside that person's range (0–4 for person 1, 20–24 for person 2). The two audio streams receive fixed labels (2 and 22), so queries from a person's region sit close to that person's audio label and far from the other's, activating only the correct part of the attention map. As a corollary, the paper claims that freezing all parameters except the audio cross-attention and audio adapter, together with a multi-task objective that includes image-to-video generation on multi-event data, is crucial for preserving the base model's instruction-following ability. The authors present experiments on talking-head, talking-body, and two-person datasets, reporting improved lip-sync and video-quality metrics over existing single-person methods.

Load-bearing premise

The method assumes the diffusion model's self-attention map reliably tracks which person each video token belongs to even under large motion, occlusion, and physical contact; if that tracking drifts, the assigned labels are wrong and the wrong audio stream binds to the wrong person.

Editorial extensions

If this is right

  • Multi-person conversational video generation is feasible in a single diffusion model, so the left and right halves no longer need to be generated separately and stitched together.
  • Because binding is label-based rather than tied to fixed screen positions, the method should handle large movements and cases where two people cross or interact physically.
  • The recipe of freezing all but the audio-conditioning layers and mixing in image-to-video data can be reused when adding other new conditions to a large video diffusion model.
  • The autoregressive extension produces long videos by conditioning on the last five frames, and the paper demonstrates a 305-frame example, making the approach usable beyond short clips.
  • The stated limitation that real audio works better than synthesized audio is a concrete challenge for future work to close.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The label-range scheme extends naturally beyond two speakers: adding a third person just requires another label interval and a person-localization step with three masks, so a three-person conversation with three audio streams would be a direct stress test.
  • Because L-RoPE acts purely in attention geometry, the same labelling idea could bind other per-subject conditions—motion embeddings, separate text prompts, or object-specific audio—to distinct subjects in multi-subject generation.
  • The paper never measures localization accuracy directly; a standalone benchmark tracking how well self-attention tokens follow a person through occlusion would reveal whether that is the fragile link, as the naive left/right baseline's failure suggests.
  • Zeroing the new conditioning input while mixing in original-task data may be a general regularization for any fine-tuning of a video foundation model, not only for audio.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces MultiTalk, a framework for audio-driven multi-person conversational video generation. It extends a DiT-based image-to-video foundation model with audio cross-attention layers, proposes Label Rotary Position Embedding (L-RoPE) to bind multiple audio streams to distinct persons, and uses reference-image-to-video self-attention maps for adaptive person localization. The training recipe combines two-stage training, partial parameter fine-tuning, and multi-task training to preserve instruction-following. The method is evaluated on talking-head datasets (HDTF, CelebV-HQ), a talking-body dataset (EMTD), and a newly collected private dual-human dataset (MTHM). The central claim is that L-RoPE resolves incorrect audio-person binding while the proposed training strategies preserve instruction-following ability.

Significance. If the central claim holds, the paper addresses a genuinely new and practically relevant task: multi-person conversational video generation with multi-stream audio. The L-RoPE mechanism is conceptually simple and potentially parameter-efficient, and the observation that partial-parameter and multi-task training preserve the base model's instruction-following ability is valuable for the community. The paper is also honest in reporting that naive injection schemes fail. However, the quantitative evidence for the binding claim is currently weak: the only binding evaluation is on a 40-video private dataset, the metrics are computed with a static left/right split that contradicts the adaptive-localization premise, and the ablation does not compare L-RoPE against the failed schemes or a no-L-RoPE baseline. The contribution is therefore defensible but not yet convincingly validated.

major comments (5)
  1. [§4.1, Appendix A.1, Table 3] The MTHM evaluation protocol computes Sync-C, Sync-D, and E-FID by splitting each generated video into left and right halves and assigning each half to one speaker's audio. This is in direct tension with Section 3.3, where the paper argues that fixed left/right splitting fails for large movement ranges and motivates adaptive person localization. Since the qualitative results include standing up, walking, and hugging, the reported binding metrics do not establish that the correct audio stream drives the correct person in dynamic interactions. The authors should either evaluate with per-frame person tracking/masks, or restrict the quantitative binding claim to clips where a fixed split is valid, and report the fraction of such clips.
  2. [§3.3, Table 3] The only quantitative ablation for L-RoPE compares two label-range choices. There is no quantitative comparison against the concatenation scheme (Fig. 3a), the additive scheme (Fig. 3b), the static left/right split (Fig. 3c), or a no-L-RoPE baseline. Consequently, the central claim that L-RoPE resolves incorrect audio-person binding is not isolated by the experiments. At minimum, the authors should report binding-aware metrics (e.g., per-person Sync-C/Sync-D computed from tracked person regions) for each injection scheme, not only for two L-RoPE variants.
  3. [§4.2, Fig. 7] The multi-person comparison is limited to a qualitative video-concatenation example. No quantitative comparison on MTHM is reported against video concatenation or any other multi-person adapted baseline, despite the abstract's claim of superior performance. Given the small private dataset (40 videos), the authors should add quantitative results and, ideally, a human evaluation of audio-person binding for all compared schemes.
  4. [§3.3, Adaptive Person Localization] Adaptive localization relies on the reference-image-to-video self-attention map of the frozen base model to assign L-RoPE labels per token. The paper does not measure the accuracy of this localization, even though incorrect labels would directly cause incorrect audio-person binding. The authors should report a localization metric (e.g., argmax-label agreement with ground-truth person masks, or token-level IoU) on MTHM, especially for the large-motion and occlusion cases that motivate the method.
  5. [§3.4, Fig. 5, Fig. 9] The claims that partial parameter training and multi-task training preserve instruction-following are supported only by qualitative examples. No quantitative instruction-following metric (e.g., success rate on action/object-interaction prompts, or CLIP-based text-video alignment) is reported. Since this is a stated core contribution, the authors should provide a quantitative evaluation or a user study.
minor comments (5)
  1. [Conclusion] The first paragraph of the conclusion says 'L-PoRE' where 'L-RoPE' is clearly meant.
  2. [§3.3, Eq. (3)-(4)] The notation for the label l_i and the normalization function is hard to parse; the inline expression 'N orm(S[i, j]j=C[person1] , a, b)' contains undefined variables and should be rewritten with a clear definition of C and min/max over the appropriate set.
  3. [Table 1] Several numeric entries are missing separators (e.g., Sonic row '8.356.431.22 29.5389.34' and MultiTalk-multiple row '32.31219.19'), making the table difficult to read.
  4. [§3.3] The method for obtaining the initial person/background masks M={M_p1, M_p2, M_b} from the reference image is not described; please specify the segmentation procedure or the source of these masks.
  5. [Appendix A.1] The MTHM dataset description gives only the number of videos; please add duration, resolution, interaction types, and data-collection/licensing details to support reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: L-RoPE is an architectural mechanism, not a fitted prediction; self-citations are non-load-bearing.

full rationale

The paper does not claim a mathematical derivation from first principles; it proposes a conditional video-generation architecture. L-RoPE is defined by Eqs. (3)-(4) as a rotary position embedding whose angle depends on a hand-assigned label, and the labels for video tokens are obtained from the frozen base model's reference-image-to-video self-attention map (Section 3.3). These label ranges are manually chosen, not fitted to Sync-C/Sync-D or any evaluation metric, so the binding behavior is a designed mechanism rather than a fitted quantity renamed as a prediction. The central capability claims are evaluated against external benchmarks (HDTF, CelebV-HQ, EMTD) using external metrics (Sync-C, Sync-D, E-FID, FID, FVD), so the results are not self-referential. The only author self-citation, OMG [29], appears in a related-work sentence listing downstream applications and is not load-bearing. The MTHM evaluation in Appendix A.1 splits generated videos into left and right halves and pairs each half with the corresponding audio; this is a validity concern because it may not track identity under large motion or crossing, but it is not circular: the fixed split does not define or fit the L-RoPE labels and the adaptive localization is not derived from the evaluation metric. Overall, no load-bearing step reduces to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a frozen pretrained 14B video diffusion model, on the reliability of self-attention-based person localization, and on several hand-chosen label hyperparameters. These are the free parameters and domain assumptions the method depends on; none are derived from first principles.

free parameters (4)
  • L-RoPE label ranges for person 1 and person 2 = [0,4] and [20,24]
    Hand-chosen ranges used to assign rotary label positions to video latents for audio-person binding. Only two variants are ablated (Table 3), and the chosen ranges are not derived from data.
  • L-RoPE static labels for background and audio streams = 12 (background), 2 (audio 1), 22 (audio 2)
    Static label values chosen by hand to separate background and bind each audio stream to its person. No systematic search is reported.
  • Audio context length k in Eq. (1) = not specified
    The window of surrounding audio frames concatenated for each video frame is a design choice that affects synchronization but is not reported.
  • RoPE base angle theta_base = not specified
    Pre-defined base angle for rotary position embedding used in Eq. (3); inherited from RoPE literature but its value for L-RoPE is not given.
assumptions (4)
  • domain assumption Wan2.1-I2V-14B provides a strong generative prior for human motion and instruction following.
    The entire framework is built on this frozen base model (Section 3.1, Implementation Details), and the paper assumes its motion and instruction-following priors are preserved under partial fine-tuning.
  • domain assumption The reference-image-to-video self-attention map reliably localizes each person across all frames of the generated video.
    Adaptive Person Localization (Section 3.3) computes S from the self-attention map and uses argmax to assign person labels. If this map fails under large motion, occlusion, or interaction, L-RoPE labels are wrong and audio binds incorrectly. The paper's own left-right split failure (Fig. 3c) shows localization is the fragile component.
  • standard math Wav2Vec audio embeddings contain sufficient speech information for lip-sync and facial expression.
    Wav2Vec is a standard pretrained audio feature extractor used widely in talking head generation (Section 3.2).
  • domain assumption DiT-based video diffusion models support additional cross-attention layers without retraining the base layers.
    The paper inserts audio cross-attention into each DiT block and freezes all other parameters; it assumes this does not destabilize the pretrained layers (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation." pith.science (2026). https://pith.science/paper/GHA2MR6O

@misc{pith2026250522647,
  author       = {Pith},
  title        = {Pith review of: Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHA2MR6O}},
  note         = {Machine review of arXiv:2505.22647}
}
read the original abstract

Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial movements and appealing visual quality videos. However, existing methods primarily focus on single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-following capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Generation, and introduce a new framework, MultiTalk, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training are crucial for preserving the instruction-following ability of the base model. MultiTalk achieves superior performance compared to other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach.

Figures

Figures reproduced from arXiv: 2505.22647 by the authors.

Figure 1
Figure 1. We propose MultiTalk, a novel framework for audio-driven multi-person conversational [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall pipeline of the proposed MultiTalk framework. Our framework incorporates [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Investigation on different injection strategies for multi-stream audio condition. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Analysis for different components in the DiT. a) We utilize the reference-image-to-video [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Instruction-following capability comparison between different training strategies. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison with other competing methods. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison with video concat method in multi-human animation. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Some examples of our MTHM dataset. A.2 Sample Details In all the experiments and evaluations conducted within this paper, we utilize 40 sampling steps. To filter out undesired variations in diffusion models, we employ the following negative prompt during sampling: "bri…
Figure 9
Figure 9. Figure 9: Comparison between full parameter training and cross-attention training. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The generation result of long videos. C Societal Impacts This paper introduces an effective tool for audio-driven multi-person conversational video generation to the community. However, there exists a risk wherein malicious entities could exploit this framework to gen…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers

    cs.CV 2025-07 conditional novelty 7.0 of 10

    A DiT-based portrait animation model transfers implicit facial expressions to one or more characters using a masked cross-attention mechanism, supported by a new multi-face dataset and benchmark.

  2. OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars

    cs.CV 2026-07 conditional novelty 6.0 of 10

    OmniMate generates real-time streaming video, speech, and sound effects for interactive avatars while preserving the same face and voice across long multi-turn conversations.

  3. Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    InterTalk is a motion-based real-time framework for flexible multi-round multi-person conversational talking face generation using motion feedback, iterative strategies, and facial component disentanglement, supported...

  4. iMontage: Unified, Versatile, Highly Dynamic Many-to-many Image Generation

    cs.CV 2025-11 conditional novelty 6.0 of 10

    iMontage repurposes a pretrained video diffusion model to generate coherent yet highly dynamic image sets from arbitrary numbers of input images.

  5. FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    A three-part system, Talking-Critic, Talking-NSQ, and TLPO, aligns diffusion portrait animation models to human preferences and improves lip-sync, motion naturalness, and visual quality.

  6. InfinityHuman: Towards Long-Term Audio-Driven Human

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A coarse-to-fine audio-driven animation framework that uses pose-guided refinement and hand-specific reward learning to generate long, identity-stable talking videos.

  7. MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation

    cs.CV 2025-08 reject novelty 5.0 of 10

    A new autoregressive video-generation framework for interactive digital humans, with a 64x compression autoencoder and a diffusion renderer, claims real-time multimodal control but is only demonstrated for audio input.

  8. InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Sparse-frame dubbing with adjacent-chunk keyframe sampling lets a streaming audio-video model produce full-body motion synchronized to new audio while preserving identity and camera motion.

Reference graph

Works this paper leans on

46 extracted references · 17 canonical work pages · cited by 8 Pith papers

  1. [1]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. InECCV, pages 244–260. Springer, 2024

    2024

  2. [2]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    arXiv 2024

  3. [3]

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer.arXiv preprint arXiv:2412.00733, 2024

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer.arXiv preprint arXiv:2412.00733, 2024

    arXiv 2024

  4. [4]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. InAAAI, volume 39, pages 2403–2410, 2025

    work page 2025

  5. [5]

    Latentsync: Audio conditioned latent diffusion models for lip sync.arXiv preprint arXiv:2412.09262, 2024

    Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. Latentsync: Audio conditioned latent diffusion models for lip sync.arXiv preprint arXiv:2412.09262, 2024

    arXiv 2024

  6. [6]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency.arXiv preprint arXiv:2409.02634, 2024

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency.arXiv preprint arXiv:2409.02634, 2024

    arXiv 2024

  7. [7]

    Cyberhost: A one-stage diffusion framework for audio-driven talking body generation

    Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, Zerong Zheng, and Yanbo Zheng. Cyberhost: A one-stage diffusion framework for audio-driven talking body generation. InICLR

    work page

  8. [8]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025

    arXiv 2025

Show all 46 references
  1. [9]

    Emo2: End-effector guided audio-driven avatar video generation.arXiv preprint arXiv:2501.10687, 2025

    Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio-driven avatar video generation.arXiv preprint arXiv:2501.10687, 2025

    2025 arXiv

  2. [10]

    Echomimicv2: Towards striking, simplified, and semi-body human animation.arXiv preprint arXiv:2411.10061, 2024

    Rang Meng, Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimicv2: Towards striking, simplified, and semi-body human animation.arXiv preprint arXiv:2411.10061, 2024

    2024

  3. [11]

    Fantasytalking: Realistic talking portrait generation via coherent motion synthesis

    Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fantasytalking: Realistic talking portrait generation via coherent motion synthesis. arXiv preprint arXiv:2504.04842, 2025

    2025 arXiv

  4. [12]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  5. [13]

    Diffusion adversarial post-training for one-step video generation.arXiv preprint arXiv:2501.08316, 2025

    Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation.arXiv preprint arXiv:2501.08316, 2025

    2025

  6. [14]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  7. [15]

    Stylesync: High-fidelity generalized and personalized lip sync in style-based generator

    Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based generator. InCVPR, pages 1505–1515, 2023

    2023

  8. [16]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. InCVPR, pages 8652–8661, 2023. 11

    2023

  9. [17]

    Videoretalking: Audio-based lip synchronization for talking head video editing in the wild

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. InACM SIGGRAPH Asia, pages 1–9, 2022

    2022

  10. [18]

    Dpe: Disentanglement of pose and expression for general video portrait editing

    Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xiaodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Disentanglement of pose and expression for general video portrait editing. InCVPR, pages 427–436, 2023

    2023

  11. [19]

    Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan

    Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. InECCV, pages 85–101. Springer, 2022

    2022

  12. [20]

    Toontalker: Cross-domain face reenactment

    Yuan Gong, Yong Zhang, Xiaodong Cun, Fei Yin, Yanbo Fan, Xuan Wang, Baoyuan Wu, and Yujiu Yang. Toontalker: Cross-domain face reenactment. InICCV, pages 7690–7700, 2023

    2023

  13. [21]

    V-express: Conditional dropout for progressive training of portrait video generation.arXiv preprint arXiv:2406.02511, 2024

    Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation.arXiv preprint arXiv:2406.02511, 2024

    2024 arXiv

  14. [22]

    Nonlinear 3d face morphable model

    Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. InCVPR, pages 7346–7355, 2018

    2018

  15. [23]

    Audio-driven dubbing for user generated contents via style-aware semi-parametric synthesis.IEEE TCSVT, 33(3):1247– 1261, 2022

    Linsen Song, Wayne Wu, Chaoyou Fu, Chen Change Loy, and Ran He. Audio-driven dubbing for user generated contents via style-aware semi-parametric synthesis.IEEE TCSVT, 33(3):1247– 1261, 2022

    2022

  16. [24]

    Aniportrait: Audio-driven synthesis of photore- alistic portrait animation.arXiv preprint arXiv:2403.17694, 2024

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photore- alistic portrait animation.arXiv preprint arXiv:2403.17694, 2024

    2024 arXiv

  17. [25]

    Sonic: Shifting focus to global audio perception in portrait animation.arXiv preprint arXiv:2411.16331, 2024

    Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonic: Shifting focus to global audio perception in portrait animation.arXiv preprint arXiv:2411.16331, 2024

    2024 arXiv

  18. [26]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InCVPR, pages 10684–10695, 2022

    2022

  19. [27]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  20. [28]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pages 22500–22510, 2023

    2023

  21. [29]

    Omg: Occlusion-friendly personalized multi-concept generation in diffusion models

    Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: Occlusion-friendly personalized multi-concept generation in diffusion models. InECCV, pages 253–270. Springer, 2024

    2024

  22. [30]

    Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023

    arXiv

  23. [31]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    2023 arXiv

  24. [32]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  25. [33]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024. 12

    2024 arXiv

  26. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, pages 4195–4205, 2023

    2023

  27. [35]

    Stereocrafter: Diffusion-based generation of long and high-fidelity stereoscopic 3d from monocular videos.arXiv preprint arXiv:2409.07447, 2024

    Sijie Zhao, Wenbo Hu, Xiaodong Cun, Yong Zhang, Xiaoyu Li, Zhe Kong, Xiangjun Gao, Muyao Niu, and Ying Shan. Stereocrafter: Diffusion-based generation of long and high-fidelity stereoscopic 3d from monocular videos.arXiv preprint arXiv:2409.07447, 2024

    2024 arXiv

  28. [36]

    Stylemas- ter: Stylize your video with artistic generation and translation.arXiv preprint arXiv:2412.07744, 2024

    Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, and Wenhan Luo. Stylemas- ter: Stylize your video with artistic generation and translation.arXiv preprint arXiv:2412.07744, 2024

    2024 arXiv

  29. [37]

    Towards multiple character image animation through enhancing implicit decoupling

    Jingyun Xue, Hongfa Wang, Qi Tian, Yue Ma, Andong Wang, Zhiyuan Zhao, Shaobo Min, Wenzhe Zhao, Kaihao Zhang, Heung-Yeung Shum, et al. Towards multiple character image animation through enhancing implicit decoupling. InICLR

  30. [38]

    Generative inbetweening: Adapting image-to-video models for keyframe interpolation.arXiv preprint arXiv:2408.15239, 2024

    Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmacher-Shlizerman, Aleksander Holyn- ski, and Steven M Seitz. Generative inbetweening: Adapting image-to-video models for keyframe interpolation.arXiv preprint arXiv:2408.15239, 2024

    2024 arXiv

  31. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763. PmLR, 2021

    2021

  32. [40]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. 33:12449–12460, 2020

    2020

  33. [41]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  34. [42]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. InCVPR, pages 3661–3670, 2021

    2021

  35. [43]

    CelebV-HQ: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. InECCV, 2022

    2022

  36. [44]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. 30, 2017

    2017

  37. [45]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019

    2019

  38. [46]

    Out of time: automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. InACCV, pages 251–263. Springer, 2017. A Dataset and Implementation Details A.1 Dataset Details In this paper, we utilize three distinct testing datasets: the talking head dataset, the talking bo...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.