Pith. sign in

REVIEW 4 major objections 5 minor 9 cited by

HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HuMo, a single model conditioned on text, reference images, and audio, claims to outperform specialized systems at both subject preservation and audio-visual sync, and to do so with one unified architecture rather than a pipeline of separat

desk verdict A solid multimodal video generation system with a useful dataset, but the 'surpasses specialized SOTA' claim overreaches on the sync task; worth refereeing with a required correction. read the letter →

arxiv 2509.08519 v1 pith:MO4C4JLB submitted 2025-09-10 cs.CV cs.MM

Liyang Chen , Tianxiang Ma , Jiawei Liu , Bingchuan Li , Zhuowei Chen , Lijie Liu , Xu He , Gen Li
show 2 more authors
Qian He Zhiyong Wu
This is my paper
classification cs.CVcs.MM
keywords human-centricvideogenerationmultimodalconditioningsubjectpreservationaudio-visualsynchronizationclassifier-freeguidancediffusiontransformerreferenceimageretrievalprogressivetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual fragmentation of human-centric video generation into separate subject-preservation and audio-driven animation pipelines is unnecessary, and that one model can learn both from text, reference images, and audio together. To overcome the scarcity of aligned triple-condition data, the authors build a dataset pipeline that pairs large-scale video with retrieved reference images and lip-synced audio. To avoid corrupting the base model's text-to-video abilities, they keep most of the diffusion backbone frozen and train only self-attention and audio-related modules in two progressive stages. The claimed payoff is flexible inference: the same model accepts text-image, text-audio, or text-image-audio inputs, while matching or beating single-task specialists on each sub-task.

What carries the argument

The load-bearing mechanism is a two-stage progressive multimodal training paradigm with three named components. The minimal-invasive image injection strategy concatenates reference-image latents at the end of the noisy video latent sequence and restricts finetuning to self-attention layers, so the model learns identity preservation without structural changes to the diffusion-transformer backbone. The focus-by-predicting strategy adds a face-mask predictor to the last four blocks, supervises it with a size-aware binary cross-entropy loss on ground-truth face masks, and treats its output as a soft regularizer that steers audio-to-face association without hard gating. The time-adaptive classifi

What would settle it

Run HuMo on the paper's audio-visual sync benchmark with the face-mask predictor disabled while keeping progressive training and time-adaptive CFG unchanged; if the sync score does not drop, the claimed sync contribution is not load-bearing. Similarly, if the released code shows the predicted face mask is never multiplied into the audio-attention weights or hidden states, the focus-by-predicting mechanism is not implemented as described.

Watch

Extended reading notes

Core claim

HuMo's central claim is that collaborative multimodal control in human video generation can be achieved by jointly designing data and training rather than by adding yet another adapter to an existing pipeline. The model takes a text prompt, one or more reference images, and optionally audio, and at inference can drop any missing modality by substituting a null token. Subject preservation is learned by appending reference-image latents to the noisy video latents and fine-tuning only the self-attention layers, which the paper says preserves the base model's prompt-following and visual-generation strengths. Audio-visual sync is added in a second training stage with audio cross-attention layers

Load-bearing premise

The claim that predicting a face region improves lip-sync rests on that prediction actually steering the network's audio-video attention, but the paper does not state how the predicted mask is applied during training or inference.

Editorial extensions

If this is right

  • A single HuMo model can generate from text-image, text-audio, or text-image-audio inputs, so users can compose conditioning modalities freely at inference without retraining.
  • Because the two stages use different data requirements, the training paradigm can exploit partly annotated datasets: subject preservation only needs video-image pairs, and audio-visual sync only needs synced audio-video pairs.
  • Keeping the base model's cross-attention and most parameters frozen preserves text following and visual quality while adding subject consistency, avoiding the degradation seen with full finetuning.
  • The 17B variant outperforms the 1.7B variant on all reported sub-tasks, indicating the multimodal training strategy transfers to larger backbones.
  • Removing the focus-by-predicting strategy hurts both audio-visual sync and identity consistency, suggesting the sync and subject-preservation objectives reinforce each other rather than purely conflicting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never specifies how the predicted face mask M_pred is applied to audio cross-attention or hidden states; if it is only supervised as a side branch, the observed sync gain may come from the auxiliary loss or from the progressive curriculum rather than from actual steering. A testable extension is to feed M_pred into the audio-attention weighting and measure the marginal improvement.
  • The decoupled two-stage curriculum suggests that new conditioning modalities could be added incrementally by appending further stages, provided partially annotated data exists for each new capability.
  • The time-adaptive CFG schedule is a hand-set two-phase rule; a per-sample learned schedule could yield finer control over the trade-off between prompt following and identity preservation.
  • The same reference-image retrieval pipeline used for humans appears applicable to objects and animals in Stage 1, so extending audio-aligned data beyond human speech could enable non-human audio-driven characters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents HuMo, a human-centric video generation framework conditioned jointly on text, reference images, and audio. It contributes three components: (i) a multimodal data processing pipeline that retrieves cross-paired reference images and filters audio-aligned clips to form triplet-conditioned training data; (ii) a progressive two-stage training paradigm in which a DiT-based T2V backbone first learns subject preservation via a minimally invasive image-injection strategy and then learns audio-visual sync through inserted audio cross-attention layers plus a proposed 'focus-by-predicting' face-mask auxiliary loss; and (iii) a time-adaptive classifier-free guidance scheme that switches modality guidance weights at different denoising steps. The authors report quantitative results on an in-house 100-case benchmark for subject preservation and on the external MoCha benchmark for audio-visual sync, and claim that HuMo surpasses specialized state-of-the-art methods on both sub-tasks.

Significance. If the empirical claims are substantiated, HuMo would be a useful contribution: a single model that accepts flexible input compositions (text-image, text-audio, text-image-audio) and performs competitively against specialized systems. The use of an external benchmark (MoCha) for the sync task and the construction of a triplet-conditioned dataset are positive elements, and the authors promise to release code, models, and data, which would aid reproducibility. However, the central 'surpasses specialized SOTA' claim is not currently supported by the reported tables: on the audio-visual sync sub-task, Ours-17B trails OmniHuman-1 on both Sync-C and Sync-D, and no statistical significance or error-bar analysis is provided for any of the quantitative comparisons. The focus-by-predicting mechanism, presented as a key technical novelty, is also under-specified. The paper's value depends on these issues being resolved.

major comments (4)
  1. [§4, Table 2 and Abstract] The abstract and conclusion state that HuMo 'surpasses specialized state-of-the-art methods in sub-tasks,' but Table 2 does not support this for audio-visual sync. Ours-17B obtains Sync-C 6.252 vs. OmniHuman-1's 6.526 and Sync-D 8.577 vs. 7.784; Ours-1.7B also trails on both sync metrics. The text concedes that the 1.7B model 'trails only slightly behind' OmniHuman-1, but the same is true of the 17B model. The headline claim must be qualified (e.g., 'surpasses open-source / text-capable specialized methods') or supported with statistical evidence and a clear definition of 'specialized SOTA.'
  2. [§3.3, Eq. (3)] The focus-by-predicting strategy is a central component for audio-visual sync, but the manuscript never specifies how the predicted mask M_pred is used. It is described as a 'soft regularizer' that 'steers the model's focus,' yet no equation or sentence states whether M_pred multiplies audio cross-attention outputs, gates hidden states, or is used only as an auxiliary BCE loss whose gradient shapes hz. If the latter is intended, the loss-weighting relative to L_FM and the inference-time role (is F_mask dropped?) must be stated. Without this, the ablation 'w/o Focus-by-Predicting' in Table 3 is difficult to interpret, and the claim that this mechanism improves audio-visual alignment is not established.
  3. [Tables 1–3] All quantitative results are single numbers with no error bars, confidence intervals, or significance tests. The margins behind several claims are small: e.g., Table 1 CLIP-I 0.687 vs. 0.677 for Phantom, and Table 3 Sync-C 6.252 vs. 5.946. The latter could be within sampling noise, especially since only one benchmark set is used. The authors should report multiple seeds/runs or bootstrap confidence intervals, and state the number of evaluated clips. For the in-house subject-preservation benchmark of 100 cases, the selection protocol (prompt construction, reference-image sourcing, human/object/animal split, exclusion criteria) is also missing, which weakens the generality of Table 1.
  4. [§4.1, subject-preservation benchmark] The in-house benchmark for the subject-preservation task is described only as 'a benchmark of 100 test cases involving humans, objects, and animals.' There is no description of how these cases were selected, how text prompts and reference images were constructed, or whether the evaluation is fully automatic. Given that Table 1 includes VLM-based metrics (HSP, TVA) and face-similarity metrics that can be sensitive to input crops, the protocol should be specified so the comparison is reproducible. Without this, the claim of SOTA subject preservation and text following is under-supported.
minor comments (5)
  1. [§3.3] Typo: 'n = 5 and and f same with the sequence length' should be 'n = 5 and f is the same as ...' Also, the audio embedding is denoted ca ∈ R^{f×n×d}, but §3 describes audio as a 'signal'; the notation mixes the raw signal and the embedded representation.
  2. [§3.3, progressive training] The curriculum schedule is vague: 'Initially ... 80% ratio ... gradually increase ... to 50%.' Over 40k steps, the exact interpolation schedule matters for reproducibility; please specify the ratio at each stage or the annealing rule.
  3. [§3.4, time-adaptive CFG] The time-adaptive CFG is claimed to 'significantly improve' collaboration, but no quantitative ablation is reported; Figure 4 is only qualitative. A table comparing static CFG configurations A/B vs. the adaptive schedule would strengthen the claim.
  4. [Table 2] OmniHuman-1 has no TVA score because its API does not accept text input; this should be stated in the table caption or main text to avoid misleading comparisons. Also, the table caption misspells 'benchmark.'
  5. [Eq. (3)] The size-aware weight is written as hw / sum M_gt · BCE(...). Please define whether h, w are spatial dimensions and whether the sum is over spatial positions; the current expression is ambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: core claims are validated against external benchmarks and independent ablations; self-citations are not load-bearing.

full rationale

I walked the claimed derivation chain and found no step where a prediction reduces by construction to a fitted input, where a parameter is fitted to a subset and then reported as a prediction of a closely related quantity, or where a load-bearing conclusion depends solely on a self-citation. The central empirical claims are evaluated against external benchmarks: subject preservation is compared on an in-house 100-case benchmark against Kling 1.6, MAGREF, HunyuanCustom, and Phantom, while audio-visual sync is evaluated on the MoCha benchmark [35] against OmniHuman-1, Hallo3, FantasyTalking, and HunyuanCustom. The data pipeline relies on external video pools [18,33], external VLMs [1,30], and external lip-sync tools [17]; self-citations to Phantom [22] and Phantom-data [3] are methodological precedents for reference-image retrieval, not evidence that establishes the headline result. The focus-by-predicting strategy (Eq. 3) is an auxiliary BCE loss on ground-truth face masks; even though the paper does not specify precisely how M_pred modulates attention or hidden states, this is an under-specified mechanism and a correctness risk, not a circular reduction to the Sync-C metric. The time-adaptive CFG is an inference-time schedule and is not derived from the paper's own outputs. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. Accordingly, the manuscript is self-contained against external evidence for its main comparisons, and I find no significant circularity.

Assumptions & free parameters 5 free parameters · 7 assumptions · 1 invented entities

The central claim rests on standard flow-matching math, a pretrained DiT backbone, a retrieval-based dataset construction pipeline, and several hand-chosen hyperparameters. The dataset alignment, the self-attention-only fine-tuning assumption, and the causal efficacy of the mask predictor are the least externally validated premises. No new physical entities are posited beyond the mask predictor module.

free parameters (5)
  • Audio temporal window length n = 5
    Chosen by hand as the number of neighboring audio frames concatenated into each frame's audio embedding (§3.3); no ablation reported for this value.
  • CFG guidance scales and time-adaptive schedules A/B = not given in paper
    §3.4 defines two CFG configurations but reports no numerical values; the switch happens at timestep 0.98. These are hand-tuned and not quantitatively ablated in any table.
  • Progressive training curriculum ratio = starts 80% subject-preservation / 20% audio-visual, ends 50/50
    §3.3 states the ratio starts at 80/20 and gradually increases the audio-visual share to 50%, but the per-step schedule is unspecified; this curriculum is central to the claimed collaborative learning.
  • Stage training durations = 40k steps (Stage 1) + 40k steps (Stage 2)
    Chosen empirically in §4 Implementation Details with no reported sweep.
  • Mask predictor placement = last 4 DiT blocks, averaged outputs
    Hand-chosen architectural detail in §3.3; no ablation for the number of blocks.
assumptions (7)
  • standard math Flow matching objective (Eq. 1) with a learned velocity field is a valid training target for video generation.
    Taken from prior work [20]; the paper does not justify it beyond citing flow matching.
  • domain assumption The Wan-2.1 DiT backbone retains its text-following and video-generation quality when only self-attention layers are updated.
    Stage 1 relies on this; the Full Finetune ablation partially supports it empirically, but the mechanism is not proven.
  • domain assumption Concatenating reference latents at the end of the video latent sequence causes the model to propagate identity via self-attention rather than treat the reference as a start frame.
    §3.3 Subject Preservation; design choice borrowed from in-context methods [7, 22], not derived from first principles.
  • domain assumption Audio embeddings in a per-frame temporal window (n=5) capture the cues needed for lip and body synchrony.
    §3.3 Audio-Visual Sync; window length is an unablated hyperparameter.
  • domain assumption Supervising a face-mask predictor with BCE loss on intermediate features guides attention toward facial regions and improves audio-visual sync.
    This is the focus-by-predicting claim itself; the paper does not specify how the predicted mask modulates attention, so the causal link is presumed.
  • domain assumption The objective metrics (VBench AES/IQA, VLM-based HSP and TVA, Face-Cur/ArcFace similarity, Sync-C/Sync-D) reflect human judgments of the stated qualities.
    Evaluation validity is assumed; no user study or calibration is provided.
  • domain assumption The retrieval and filtering pipeline yields reference images that share identity with the video subject while differing in appearance, and audio tracks with reliable lip alignment.
    Data pipeline in §3.2; failures here would directly degrade subject preservation and sync.
invented entities (1)
  • F_mask (face location mask predictor)
    purpose: Estimate facial region distribution from DiT hidden states to implicitly steer audio-visual sync.
    Introduced in §3.3. No external validation; its effectiveness rests solely on the internal Table 3 ablation, and the mechanism by which it modulates generation is not specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning." pith.science (2026). https://pith.science/paper/MO4C4JLB

@misc{pith2026250908519,
  author       = {Pith},
  title        = {Pith review of: HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MO4C4JLB}},
  note         = {Machine review of arXiv:2509.08519}
}
read the original abstract

Human-Centric Video Generation (HCVG) methods seek to synthesize human videos from multimodal inputs, including text, image, and audio. Existing methods struggle to effectively coordinate these heterogeneous modalities due to two challenges: the scarcity of training data with paired triplet conditions and the difficulty of collaborating the sub-tasks of subject preservation and audio-visual sync with multimodal inputs. In this work, we present HuMo, a unified HCVG framework for collaborative multimodal control. For the first challenge, we construct a high-quality dataset with diverse and paired text, reference images, and audio. For the second challenge, we propose a two-stage progressive multimodal training paradigm with task-specific strategies. For the subject preservation task, to maintain the prompt following and visual generation abilities of the foundation model, we adopt the minimal-invasive image injection strategy. For the audio-visual sync task, besides the commonly adopted audio cross-attention layer, we propose a focus-by-predicting strategy that implicitly guides the model to associate audio with facial regions. For joint learning of controllabilities across multimodal inputs, building on previously acquired capabilities, we progressively incorporate the audio-visual sync task. During inference, for flexible and fine-grained multimodal control, we design a time-adaptive Classifier-Free Guidance strategy that dynamically adjusts guidance weights across denoising steps. Extensive experimental results demonstrate that HuMo surpasses specialized state-of-the-art methods in sub-tasks, establishing a unified framework for collaborative multimodal-conditioned HCVG. Project Page: https://phantom-video.github.io/HuMo.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What if? Emulative Simulation with World Models for Situated Reasoning

    cs.CV 2026-03 conditional novelty 6.5 of 10

    WanderDream supplies 15.8K panoramic mental-exploration videos and 158K QA pairs showing that world-model imagination measurably improves situated spatial reasoning without active exploration.

  2. ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A unified video model with role-aware context and fixed-budget sparse attention enables interactive multi-shot generation, reference conditioning, and editing at 16 FPS on a single GPU.

  3. ID-V2V: Identity-Preserving Video Restylization

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ID-V2V restyles video by conditioning a diffusion model on edited keyframes, depth, relit faces, and face normals, so scene edits propagate while facial identity and performance are preserved.

  4. AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    AgentHOI generates human-object interaction videos from text plus one human image and one object image, using multi-agent action planning and implicit text-to-motion feature alignment inside a video diffusion model.

  5. StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A streaming image-to-video model that assigns block-specific sink/local memory and learned RoPE distance scaling preserves long human-object interactions under bounded latency.

  6. Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Aura combines VLM meta-queries, T5-teacher alignment, subject-aware RoPE shifts, memory tokens, and a large AIGC-curated dataset to claim SOTA multi-element subject-to-video generation under OpenS2V-Eval Total score.

  7. MVHOI: Bridge Multi-view Condition to Complex Human-Object Interaction Video Reenactment via 3D Foundation Model

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Using a 3D foundation model to produce viewpoint-aware anchors plus multi-view reference textures enables realistic human-object-interaction reenactment with large out-of-plane rotations.

  8. OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model

    cs.SD 2026-02 conditional novelty 6.0 of 10

    A zero-shot model that generates a video of a reference face speaking user-chosen text with a reference voice timbre.

  9. Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interaction

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A distilled video model jointly controls multi-scale upper-body motion latents and two discrete contact states to generate real-time human–object interaction at 25 FPS.

Reference graph

Works this paper leans on

39 extracted references · 7 linked inside Pith · cited by 9 Pith papers

  1. [1]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and et al. Qwen2.5-vl technical report, 2025

    2025

  2. [2]

    Goku: Flow based video generative foundation models.arXiv preprint arXiv:2502.04896, 2025

    Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, and et al. Goku: Flow based video generative foundation models.arXiv preprint arXiv:2502.04896, 2025

    arXiv 2025

  3. [3]

    Phantom-data : Towards a general subject-consistent video generation dataset, 2025

    Zhuowei Chen, Bingchuan Li, Tianxiang Ma, Lijie Liu, Mingcong Liu, Yi Zhang, Gen Li, Xinghui Li, Siyu Zhou, Qian He, and Xinglong Wu. Phantom-data : Towards a general subject-consistent video generation dataset, 2025

    2025

  4. [4]

    Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining

    Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, and Noah Con- stant. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=kXwdL1cWOAi

    2023

  5. [5]

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21086–21095, June 2025

    2025

  6. [6]

    Arcface: Additive angular margin loss for deep face recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1–1, 2021

    Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Cotsia, and Stefanos P Zafeiriou. Arcface: Additive angular margin loss for deep face recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1–1, 2021

    2021

  7. [7]

    Magref: Masked guidance for any-reference video generation, 2025

    Yufan Deng, Xun Guo, Yuanyang Yin, Jacob Zhiyuan Fang, Yiding Yang, Yizhi Wang, Shenghai Yuan, Angtian Wang, Bo Liu, Haibin Huang, and Chongyang Ma. Magref: Masked guidance for any-reference video generation, 2025

    2025

  8. [8]

    Seedream 3.0 technical report, 2025

    Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, and et al. Seedream 3.0 technical report, 2025

    2025

Show all 39 references
  1. [9]

    Seedance 1.0: Exploring the boundaries of video generation models, 2025

    Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, and et al. Seedance 1.0: Exploring the boundaries of video generation models, 2025. URL https://arxiv.org/abs/2506.09113

    2025 arXiv

  2. [10]

    Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024

    Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, and Jie Zhang. Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024

    2024 arXiv

  3. [11]

    Hunyuancustom: A multimodal-driven architecture for customized video generation, 2025

    Teng Hu, Zhentao Yu, Zhengguang Zhou, Sen Liang, Yuan Zhou, Qin Lin, and Qinglin Lu. Hunyuancustom: A multimodal-driven architecture for customized video generation, 2025

    2025

  4. [12]

    Curricularface: Adaptive curriculum learning loss for deep face recognition

    Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: Adaptive curriculum learning loss for deep face recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020

    2020

  5. [13]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models. I...

    2024

  6. [14]

    Multi-reference images to video generation feature

    Kling. Multi-reference images to video generation feature. https://app.klingai.com/cn/release-notes/ 2025-07-24, 2025

    2025

  7. [15]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  8. [16]

    Let them talk: Audio-driven multi-person conversational video generation, 2025

    Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conversational video generation, 2025

    2025

  9. [17]

    Latentsync: Taming audio-conditioned latent diffusion models for lip sync with syncnet supervision,

    Chunyu Li, Chao Zhang, Weikai Xu, Jingyu Lin, Jinghui Xie, Weiguo Feng, Bingyue Peng, Cunjian Chen, and Weiwei Xing. Latentsync: Taming audio-conditioned latent diffusion models for lip sync with syncnet supervision,

  10. [18]

    Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation

    Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, and Siyu Zhu. Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation. InProceedings of the IEEE/CVF Conference on Computer Visio...

    2025

  11. [19]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  12. [20]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t

    2023

  13. [21]

    Improving video generation with human feedback, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, and Wanli Ouyang. Improving video generation with human feedback, 2025

    2025

  14. [22]

    Phantom: Subject-consistent video generation via cross-modal alignment

    Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Jiawei Liu, Gen Li, Siyu Zhou, Qian He, and Xinglong Wu. Phantom: Subject-consistent video generation via cross-modal alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  15. [23]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, page 38–55, 2024

    2024

  16. [24]

    Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the...

    2021

  18. [26]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InProceedings of the 40th International Conference on Machine Learning, volume 202, pages 28492–28518, 2023

    2023

  19. [27]

    Seaweed-7b: Cost-effective training of video generation foundation model, 2025

    Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, and et al. Seaweed-7b: Cost-effective training of video generation foundation model, 2025. URL https://arxiv.org/abs/2504.08685

    2025 arXiv

  20. [28]

    Seitz, and Ira Kemelmacher-Shlizerman

    Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learning lip sync from audio.ACM Trans. Graph., 36(4), 2017

    2017

  21. [29]

    Gemini 2.5 flash image

    Gemini Team. Gemini 2.5 flash image. https://aistudio.google.com/prompts/new_chat?model=gemini-2. 5-flash-image-preview/, 2025

    2025

  22. [30]

    Gemini 2.5: Our most intelligent ai model.https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/, 2025

    Gemini Team. Gemini 2.5: Our most intelligent ai model.https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/, 2025

    2025

  23. [31]

    Wan: Open and advanced large-scale video generative models, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, and et al. Wan: Open and advanced large-scale video generative models, 2025

    2025

  24. [32]

    Fantasytalking: Realistic talking portrait generation via coherent motion synthesis

    Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fantasytalking: Realistic talking portrait generation via coherent motion synthesis. InProceedings of the 33th ACM International Conference on Multimedia, 2025

    2025

  25. [33]

    Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content

    Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, Fei Yang, Pengfei Wan, and Di Zhang. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. InProceed...

    2025

  26. [34]

    Interacthuman: Multi-concept human animation with layout-aligned audio conditions, 2025

    Zhenzhi Wang, Jiaqi Yang, Jianwen Jiang, Chao Liang, Gaojie Lin, Zerong Zheng, Ceyuan Yang, and Dahua Lin. Interacthuman: Multi-concept human animation with layout-aligned audio conditions, 2025. 14

    2025

  27. [35]

    Mocha: Towards movie-grade talking character synthesis, 2025

    Cong Wei, Bo Sun, Haoyu Ma, Ji Hou, Felix Juefei-Xu, Zecheng He, Xiaoliang Dai, Luxin Zhang, Kunpeng Li, Tingbo Hou, Animesh Sinha, Peter Vajda, and Wenhu Chen. Mocha: Towards movie-grade talking character synthesis, 2025

    2025

  28. [36]

    Magicinfinite: Generating infinite talking videos with your words and voice, 2025

    Hongwei Yi, Tian Ye, Shitong Shao, Xuancheng Yang, Jiantong Zhao, Hanzhong Guo, Terrance Wang, Qingyu Yin, Zeke Xie, Lei Zhu, Wei Li, Michael Lingelbach, and Daquan Zhou. Magicinfinite: Generating infinite talking videos with your words and voice, 2025

    2025

  29. [37]

    Opens2v-nexus: A detailed benchmark and million-scale dataset for subject-to-video generation, 2025

    Shenghai Yuan, Xianyi He, Yufan Deng, Yang Ye, Jinfa Huang, Bin Lin, Jiebo Luo, and Li Yuan. Opens2v-nexus: A detailed benchmark and million-scale dataset for subject-to-video generation, 2025

    2025

  30. [38]

    Identity- preserving text-to-video generation by frequency decomposition

    Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyang Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. Identity- preserving text-to-video generation by frequency decomposition. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 12978–12988, 2025. 15

    2025

  31. [2025]

    URLhttps://arxiv.org/abs/2412.09262. 13

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.