Pith. sign in

REVIEW 3 major objections 5 minor 87 references

Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two-stage training on noisy auto-curated videos and a few clean examples scales audio-synchronized visual animation to open classes, cutting manual curation by over 10x.

desk verdict Solid scaling recipe for audio-synchronized animation, but the 10x-curation claim has a likely train-set leakage problem that needs to be answered. read the letter →

arxiv 2508.03955 v1 pith:RGXQYYSF submitted 2025-08-05 cs.CV

Lin Zhang , Zefan Cai , Yufan Zhou , Shentong Mo , Jinhong Lin , Cheng-En Wu , Yibing Wei , Yijing Zhang
show 5 more authors
Ruiyi Zhang Wen Xiao Tong Sun Junjie Hu Pedro Morgado
This is my paper
classification cs.CV
keywords audio-synchronizedvisualanimationtwo-stagetrainingfew-shotfine-tuningdiffusionvideogenerationaudioconditioningwindowattentionVGGSoundAVSync48
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a two-stage training paradigm for audio-synchronized visual animation: pretrain on 152K automatically filtered VGGSound videos that still contain many imperfect audio-visual alignments, then fine-tune on a small set of manually curated high-quality videos. The central claim is that the noisy pretraining stage teaches a weak but transferable synchronization signal, so one curated video per class after pretraining outperforms ten curated videos per class without it, cutting manual curation by over 10x. To keep this affordable, the authors freeze a pretrained text-to-video generator and two audio encoders, training only lightweight audio-conditioning layers that add 1.9% parameters. They also introduce AVSync48, a 48-class benchmark, and report near-ceiling synchronization scores on the existing AVSync15 benchmark with a 12-FPS model. If the claim holds, the main obstacle to scaling audio-driven animation to open classes, expensive human curation, is no longer a barrier.

What carries the argument

The load-bearing mechanism is a two-stage quantity-quality trade: automatic curation for scale, human curation for precision. On the model side, the central component is a set of lightweight audio cross-attention layers inserted before temporal attention in a frozen text-to-video diffusion model; each generated frame attends to audio tokens from a local window of radius r = 1.5/FPS, sharing some tokens with neighboring frames instead of the rigid one-chunk-per-frame mapping used in prior work. The audio tokens are multi-feature projections of ImageBind and BEATs representations drawn from low, mid, and high layers, so conditioning combines semantic grounding with fine-grained rhythmic cues.

What would settle it

Take the same stage-2 fine-tuning with K=1 curated video per class, and compare three stage-1 pretraining conditions: matched VGGSound audio-video pairs, audio swapped across unrelated videos, and audio shifted by a fixed random offset. If the swapped or shifted pretraining produces the same AlignSync gains as matched pretraining on AVSync15, then the weak alignment signal is not load-bearing and the 10x curation reduction would come from data diversity alone rather than from learning synchronization.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that weakly aligned audio-visual data at scale and a few precisely aligned examples are complements, not substitutes. Stage one automatically removes only visual noise (scene cuts, sharp motion, text overlays) from 152K VGGSound videos, leaving many ambiguous audio-visual pairings; stage two fine-tunes on a hand-verified handful. The empirical crux is the few-shot comparison: with K=1 curated video per class after pretraining, the model reaches AlignSync 94.28 on AVSync15 and 94.72 on AVSync48, beating fine-tuning-only baselines trained with K=10 curated videos per class (93.32 and 94.16, respectively). At the full K=90 budget the pretrained model s

Load-bearing premise

The load-bearing premise is that the 152K automatically filtered VGGSound videos, despite containing many imperfect or ambiguous audio-visual alignments, teach a weak synchronization signal that transfers to clean few-shot fine-tuning rather than teaching systematic misalignment.

Editorial extensions

If this is right

  • Manual curation per class can drop from roughly 90 videos to about 1 video while keeping or improving synchronization metrics, making expansion to new audio-visual classes annotation-feasible.
  • The same 12-FPS model, after K=10 fine-tuning on AVSync48, animates classes never seen in the benchmark (eating, walking, slapping, volcano sounds) with plausible audio-locked motion.
  • A model with only 1.9% trainable parameters outperforms prior ASVA methods on both FVD and synchronization metrics, and the 12-FPS version approaches the ground-truth ceiling on AVSync15.
  • AVSync48 triples the class diversity of prior evaluation benchmarks, giving future scaling studies a broader testbed.
  • Automatic curation filters matter: pretraining on raw, unfiltered VGGSound is measurably worse than pretraining on the filtered 152K set on both FVD and AlignSync.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested consequence of the paper's logic is that the same noisy-pretrain plus few-shot-clean recipe could cut annotation for other audio-visual tasks with abundant weak alignment and scarce precise alignment, such as speech-driven gesture or music performance animation.
  • The near-ceiling RelSync 99.18 on AVSync15 suggests the benchmark may be saturating; distinguishing future models may require finer temporal probes such as phase-shifted audio or event-onset accuracy, which the paper does not report.
  • The curation filters are described functionally but without thresholds; a direct test of the paradigm's reproducibility would be to run the same pipeline on another large corpus and check whether the K=1-beats-K=10 pattern persists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a two-stage training paradigm for audio-synchronized visual animation (ASVA). Stage 1 pretrains on 152K automatically filtered VGGSound videos; Stage 2 fine-tunes on a small number of manually curated high-quality videos. The method freezes DynamiCrafter, ImageBind, and BEATs and introduces lightweight multi-feature audio conditioning and window-based audio cross-attention (about 1.9% trainable parameters). The paper also introduces AVSync48, a 48-class benchmark, and reports state-of-the-art results on AVSync15 and AVSync48, including few-shot experiments claiming a >10x reduction in manual curation effort.

Significance. If the claims hold, this is a practical contribution: it offers a way to scale ASVA beyond small manually curated domains, introduces a more diverse benchmark, and shows that a frozen text-to-video generator can be adapted to audio with very few trainable parameters. The paper is empirically rich: it includes ablations of the curation pipeline, fine-tuning budget, audio feature layers, and window radius, and it reports on two benchmarks. The 12-FPS model's synchronization scores near the ground-truth bound on AVSync15 are impressive. The main risk is that the headline few-shot/curation-reduction result may be confounded by overlap between the stage-1 pretraining pool and the AVSync15/AVSync48 training videos, as detailed below.

major comments (3)
  1. [§3, 'Data curation for ASVA pretraining' and Table 3] The paper states that the auto-curated VGGSound pool was built 'while also ensuring no overlap with AVSync15 and AVSync48 test sets,' but it does not exclude the AVSync15 or AVSync48 training splits. AVSync48 is itself derived from VGGSound and AudioSet, and the earlier AVSync15 is also a VGGSound-derived benchmark. It is therefore likely that the 10 training videos per class used for K-shot fine-tuning are already present in the 152K pretraining set, or are near-duplicates of those clips. Table 3 then compares pretraining + K=1 fine-tuning against fine-tuning-only with K=10; if the K=1 (or K=10) videos were seen in stage 1, the comparison conflates genuine transfer from noisy video with direct memorization of the target-domain clips. This is load-bearing for the central '>10x manual-curation reduction' claim. Please explicitly remove AVSync15 and AVSync48 training (and validation) split
  2. [§3, 'Data curation for ASVA pretraining'] The automatic curation pipeline is described only qualitatively: 'too short,' 'low FPS or resolution,' 'overwhelmingly sharp motions,' 'overwhelming onscreen text' are not quantified, and no thresholds are given for PySceneDetect, RAFT optical flow, or CRAFT text detection. The stage-1 premise—that weakly aligned noisy videos provide a learnable synchronization signal—depends on the exact filtering regime. Without thresholds, implementation details, or a released dataset with clip-level metadata, the pretraining data cannot be independently reconstructed or audited. Please provide concrete thresholds and, if possible, release the filtering code and the resulting video IDs.
  3. [§5.1 and Figure 4] The window radius (r = 1.5/FPS) and audio feature layers (3, 7, 11) are selected via ablations on AVSync15 (Figure 4), and the same benchmark is then used for the headline results in Table 1. If AVSync15 test videos were used for model selection, the reported numbers are optimistically biased. Please clarify whether a held-out validation split was used, or re-report Table 1 with the selected configuration evaluated on a split not used for ablation. This does not invalidate the comparison on AVSync48, but it affects the strength of the AVSync15 claims.
minor comments (5)
  1. [Abstract and §4] The trainable-parameter fraction is stated as 1.9% in the abstract and in §4's '55M parameters, or ... 1.9% of the overall model,' but Figure 3(a) says '1.8%.' Please reconcile.
  2. [Table 3 caption] The sentence 'Superscripts shows impact of the pretraining stage. Superscripts denote performance gains or losses.' is duplicated and awkwardly phrased. Also, the superscript arrows in the table are not explained in the caption.
  3. [§5.2] The sentence 'At 6-FPS, our model surpasses previous methods significantly on all metrics on both AVSync15 and AVSync48' should be checked against Table 1: on AVSync15, KeyVID has higher RelSync/AlignSync (96.66/96.32 vs 94.66/94.48 for Ours 6-FPS), which is acknowledged two sentences later. Please soften the claim or explicitly exclude KeyVID.
  4. [§5.3, 'Open domain video generation'] Minor typo: 'pretraind' should be 'pretrained.' Also 'man open classes' in §2.3 should be 'many open classes.'
  5. [§5.1, evaluation metrics] The paper rescales RelSync and AlignSync to 0-100 but does not state whether the rescaling changes the statistical properties of the metrics or whether scores can exceed 100. A brief clarification would help.

Circularity Check

1 steps flagged · score 5.0 of 10

Pretraining pool may contain the K=1 benchmark videos, making the '10x curation reduction' claim circular.

  1. other [Section 3, 'Data curation for ASVA pretraining' and 'AVSync48 benchmark'; Table 3]
    "In practice, we deployed the above curation steps on VGGSound [11](170K videos), while also ensuring no overlap with AVSync15 and AVSync48 test sets. ... To validate our training paradigm, we manually curated a benchmark consisting of 48 categories of diverse audio-video motions derived from VGGSound [11] and AudioSet [22]."

    By the paper's own curation description, the only overlap excluded from the 152K VGGSound pretraining pool is the test sets of AVSync15/AVSync48. Since AVSync48 is explicitly derived from VGGSound (and AudioSet), its 10-video-per-class training split is not stated to be removed from the pretraining pool. Table 3's central claim—'pretrain+K=1' beats 'finetune-only K=10'—therefore may be measuring a model that has already been trained on the very K=1 clips at scale. In that case the 'over 10x' reduction in manual curation is not a genuine few-shot result but a direct consequence of including those videos in stage-1 pretraining. The paper neither releases a data sheet nor documents an exclusion step for training splits, so this reduction is not independently established.

full rationale

The paper is primarily an empirical systems contribution: the two-stage training paradigm, audio multi-feature conditioning, and window attention are evaluated against prior work and ablations. The architecture and training details are not circular; they do not derive outputs from fitted constants in the sense of a closed-form reduction. The main circularity concern is data overlap: the pretraining pool is VGGSound with only test-set overlap excluded, while the evaluation and fine-tuning benchmarks (AVSync48, and likely AVSync15) are themselves derived from VGGSound. If the K=1 fine-tuning videos are already in the pretraining set, the headline claim 'over 10x reduction in manual curation' reduces to a form of training-data leakage rather than a genuine few-shot gain. The paper's own text conspicuously excludes only test sets, not training splits, so this confound is left unresolved. The RelSync/AlignSync metrics come from prior work by the same group, but they are externally defined benchmark scores and are corroborated by FVD and CLIP similarity, so they do not independently raise the circularity score. Overall, the empirical results are largely self-contained, but the central '10x' claim is seriously undermined by the missing train-split exclusion, giving a moderate-to-substantial circularity score of 5.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; it introduces a dataset artifact (AVSync48) and a training/architecture recipe. The listed free parameters are hyperparameters and curation thresholds that are selected by hand or by ablation on the evaluation benchmarks. The axioms are the domain assumptions underlying the two-stage training claim and the self-defined evaluation metrics.

free parameters (4)
  • Audio window radius r = 1.5/FPS
    Selected on AVSync15 by ablation in Fig. 4b; values from 0.5/FPS to 6/FPS were tested and 1.5/FPS gave the highest RelSync. Final reported results use this best radius, so AVSync15 scores partly reflect test-set selection.
  • Audio feature layers = ImageBind and BEATs layers {3,7,11}
    Chosen by ablation on AVSync15 in Fig. 4a; alternative layer sets scored lower on RelSync. This is an architectural hyperparameter selected on the evaluation benchmark.
  • Classifier-free guidance scales = image 2.0, text 2.0, audio 4.0
    Set by hand in Section 5.1; the audio guidance scale directly controls the strength of audio conditioning and is not ablated in the paper.
  • Automatic curation thresholds = not specified
    The pipeline removes short, low-FPS, low-resolution videos, scene cuts, strong optical flow, and heavy text, but exact threshold values are not given. These thresholds determine the 152K pretraining set and therefore the pretraining signal.
assumptions (5)
  • domain assumption Large-scale automatically filtered videos from VGGSound provide transferable weak audio-visual synchronization signals despite imperfect alignment.
    Stage-1 pretraining in Section 3 is built on this premise. Tables 2 and 3 support it empirically, but the '10x reduction' claim collapses if noisy pretraining does not transfer to clean finetuning.
  • domain assumption RelSync and AlignSync scores reflect audible-visual synchronization quality.
    These metrics, defined in prior work by the same group [83], are the headline synchronization measures. No human validation of the metrics is provided in this preprint.
  • domain assumption Freezing DynamiCrafter, ImageBind, and BEATs preserves their priors while audio cross-attention can add synchronization capability.
    The architecture in Section 4 trains only projection and audio cross-attention layers. If the frozen features lack temporal alignment information, the method cannot succeed.
  • domain assumption Window attention with radius 1.5/FPS provides the right specificity-coverage trade-off while other temporal layers remain adequate.
    Chosen by ablation in Fig. 4b. This is an architectural premise rather than a derivational result, and it was selected on the same benchmark used for headline results.
  • standard math Latent diffusion training and DDIM sampling provide valid video generation and inference for this model.
    The method inherits DynamiCrafter's diffusion training and DDIM inference without new proofs. This is standard background that the paper relies on.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm." pith.science (2026). https://pith.science/paper/RGXQYYSF

@misc{pith2026250803955,
  author       = {Pith},
  title        = {Pith review of: Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RGXQYYSF}},
  note         = {Machine review of arXiv:2508.03955}
}
abstract

Recent advances in audio-synchronized visual animation enable control of video content using audios from specific classes. However, existing methods rely heavily on expensive manual curation of high-quality, class-specific training videos, posing challenges to scaling up to diverse audio-video classes in the open world. In this work, we propose an efficient two-stage training paradigm to scale up audio-synchronized visual animation using abundant but noisy videos. In stage one, we automatically curate large-scale videos for pretraining, allowing the model to learn diverse but imperfect audio-video alignments. In stage two, we finetune the model on manually curated high-quality examples, but only at a small scale, significantly reducing the required human effort. We further enhance synchronization by allowing each frame to access rich audio context via multi-feature conditioning and window attention. To efficiently train the model, we leverage pretrained text-to-video generator and audio encoders, introducing only 1.9\% additional trainable parameters to learn audio-conditioning capability without compromising the generator's prior knowledge. For evaluation, we introduce AVSync48, a benchmark with videos from 48 classes, which is 3$\times$ more diverse than previous benchmarks. Extensive experiments show that our method significantly reduces reliance on manual curation by over 10$\times$, while generalizing to many open classes.

Figures

Figures reproduced from arXiv: 2508.03955 by the authors.

Figure 1
Figure 1. Generation results on open-domain images, captions, and audios from internet. Out-of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data collection and training paradigm. We automatically curate video datasets to obtain [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Architecture overview. We build our method on a frozen pretrained text guided [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation on proposed audio conditioning modules on AVSync15. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

87 extracted references · 44 canonical work pages

  1. [1]

    https://github.com/christophschuhmann/ improved-aesthetic-predictor

    Laion-aesthetics predictor v2. https://github.com/christophschuhmann/ improved-aesthetic-predictor

  2. [2]

    https://support.apple.com/en-us/106375

    Quicktime player. https://support.apple.com/en-us/106375

  3. [3]

    AI@Meta: Llama 3 model card (2024), https://github.com/meta-llama/llama3/blob/main/ MODEL_CARD.md

    2024

  4. [4]

    In: Advances in Neural Information Processing Systems (2016)

    Aytar, Y ., V ondrick, C., Torralba, A.: Soundnet: Learning sound representations from unlabeled video. In: Advances in Neural Information Processing Systems (2016)

    2016

  5. [5]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Baek, Y ., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9365–9374 (2019)

    2019

  6. [6]

    In: IEEE International Conference on Computer Vision (2021)

    Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: IEEE International Conference on Computer Vision (2021)

    2021

  7. [7]

    Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.: Mixmatch: A holistic approach to semi-supervised learning (2019), https://arxiv.org/abs/1905.02249

    arXiv 2019

  8. [8]

    https://www.scenedetect.com/

    Castellano, B.: Pyscenedetect. https://www.scenedetect.com/

Show all 87 references
  1. [9]

    Chen, H., Xia, M., He, Y ., Zhang, Y ., Cun, X., Yang, S., Xing, J., Liu, Y ., Chen, Q., Wang, X., Weng, C., Shan, Y .: Videocrafter1: Open diffusion models for high-quality video generation (2023)

    2023

  2. [10]

    In: Proceedings of the British Machine Vision Conference (BMVC) (2021)

    Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., Zisserman, A.: Audio-visual synchronization in the wild. In: Proceedings of the British Machine Vision Conference (BMVC) (2021)

    2021

  3. [11]

    In: ICASSP (2020)

    Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio-visual dataset. In: ICASSP (2020)

    2020

  4. [12]

    In: ICML (2023)

    Chen, S., Wu, Y ., Wang, C., Liu, S., Tompkins, D., Chen, Z., Wei, F.: Beats: Audio pre-training with acoustic tokenizers. In: ICML (2023)

    2023

  5. [13]

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations (2020), https://arxiv.org/abs/2002.05709

    2020 arXiv

  6. [14]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Chen, T.S., Siarohin, A., Menapace, W., Deyneka, E., Chao, H.w., Jeon, B.E., Fang, Y ., Lee, H.Y ., Ren, J., Yang, M.H., Tulyakov, S.: Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  7. [15]

    arXiv preprint arXiv:2003.04297 (2020)

    Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)

    2003 arXiv

  8. [16]

    Chiang, W.L., Li, Z., Lin, Z., Sheng, Y ., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y ., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (March 2023), https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  9. [17]

    In: ACCV (2016)

    Chung, J.S., Zisserman, A.: Lip reading in the wild. In: ACCV (2016)

    2016

  10. [18]

    In: ACCV Workshop (2016)

    Chung, J.S., Zisserman, A.: Out of time: automated lip sync in the wild. In: ACCV Workshop (2016)

    2016

  11. [19]

    In: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2019)

    Chung, S.W., Chung, J.S., Kang, H.G.: Perfect match: Improved cross-modal embeddings for audio-visual synchronisation. In: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2019)

    2019

  12. [20]

    org/abs/2401.11605

    Crowson, K., Baumann, S.A., Birch, A., Abraham, T.M., Kaplan, D.Z., Shippole, E.: Scalable high- resolution pixel-space image synthesis with hourglass diffusion transformers (2024), https://arxiv. org/abs/2401.11605

    2024

  13. [21]

    In: ICASSP (2021)

    Ebenezer, J.P., Wu, Y ., Wei, H., Sethuraman, S., Liu, Z.: Detection of audio-video synchronization errors via event detection. In: ICASSP (2021)

    2021

  14. [22]

    In: Proc

    Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., Ritter, M.: Audio set: An ontology and human-labeled dataset for audio events. In: Proc. IEEE ICASSP 2017 (2017)

    2017

  15. [23]

    In: CVPR (2023) 10

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V ., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: CVPR (2023) 10

    2023

  16. [24]

    In: Saul, L., Weiss, Y ., Bottou, L

    Grandvalet, Y ., Bengio, Y .: Semi-supervised learning by entropy minimization. In: Saul, L., Weiss, Y ., Bottou, L. (eds.) Advances in Neural Information Processing Systems. vol. 17. MIT Press (2004), https://proceedings.neurips.cc/paper_files/paper/2004/file/ 96f2b50b5d3613a...

    2004

  17. [25]

    arXiv preprint arXiv:2501.00103 (2024)

    HaCohen, Y ., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., Levin, E., Shiran, G., Zabari, N., Gordon, O., Panet, P., Weissbuch, S., Kulikov, V ., Bitterman, Y ., Melumian, Z., Bibi, O.: Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501...

    2024 arXiv

  18. [27]

    He, K., Chen, X., Xie, S., Li, Y ., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners (2021), https://arxiv.org/abs/2111.06377

    2021 arXiv

  19. [28]

    arXiv preprint arXiv:1911.05722 (2019)

    He, K., Fan, H., Wu, Y ., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 (2019)

    1911 arXiv

  20. [29]

    In: NeurIPS (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)

    2020

  21. [30]

    Hu, E.J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models (2021), https://arxiv.org/abs/2106.09685

    2021 arXiv

  22. [31]

    In: NeurIPS (2022)

    Huang, P.Y ., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., Feichtenhofer, C.: Masked autoencoders that listen. In: NeurIPS (2022)

    2022

  23. [32]

    In: British Machine Vision Conference (BMVC) (2022)

    Iashin, V ., Xie, W., Rahtu, E., Zisserman, A.: Sparse in space and time: Audio-visual synchronisation with trainable selectors. In: British Machine Vision Conference (BMVC) (2022)

    2022

  24. [33]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Jeong, Y ., Ryoo, W., Lee, S., Seo, D., Byeon, W., Kim, S., Kim, J.: The power of sound (tpos): Audio reactive video generation with stable diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7822–7832 (2023)

    2023

  25. [34]

    In: ICCV (2023)

    Khachatryan, L., Movsisyan, A., Tadevosyan, V ., Henschel, R., Wang, Z., Navasardyan, S., Shi, H.: Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In: ICCV (2023)

    2023

  26. [35]

    In: NAACL-HLT (2019)

    Kim, C.D., Kim, B., Lee, H., Kim, G.: Audiocaps: Generating captions for audios in the wild. In: NAACL-HLT (2019)

    2019

  27. [36]

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017), https://arxiv.org/abs/ 1412.6980

    2017 arXiv

  28. [37]

    semanticscholar.org/CorpusID:13874643

    Koch, G.R.: Siamese neural networks for one-shot image recognition (2015), https://api. semanticscholar.org/CorpusID:13874643

    2015

  29. [38]

    arXiv preprint arXiv:2412.03603 (2024)

    Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al.: Hun- yuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024)

    2024 arXiv

  30. [39]

    In: Advances in Neural Information Processing Systems (2018)

    Korbar, B., Tran, D., Torresani, L.: Cooperative learning of audio and video models from self-supervised synchronization. In: Advances in Neural Information Processing Systems (2018)

    2018

  31. [40]

    Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning (2017), https://arxiv.org/ abs/1610.02242

    2017 arXiv

  32. [41]

    In: ECCV (2022)

    Lee, S.H., Oh, G., Byeon, W., Bae, J., Kim, C., Ryoo, W.J., Yoon, S.H., Kim, J., Kim, S.: Sound-guided semantic video generation. In: ECCV (2022)

    2022

  33. [42]

    In: CVPR Workshop on Content Generation (2023)

    Lee, S., Kong, C., Jeon, D., Kwak, N.: Aadiff: Audio-aligned video synthesis with text-to-image diffusion. In: CVPR Workshop on Content Generation (2023)

    2023

  34. [43]

    In: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

    Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 3–12. SIGIR ’94, Springer-Verlag, Berlin, Heidelberg (1994)

    1994

  35. [44]

    In: ICCV (2021)

    Li, R., Yang, S., Ross, D.A., Kanazawa, A.: Learn to dance with aist++: Music conditioned 3d dance generation. In: ICCV (2021)

    2021

  36. [45]

    In: CVPR (2023) 11

    Li, Y ., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y .J.: Gligen: Open-set grounded text-to-image generation. In: CVPR (2023) 11

    2023

  37. [46]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Liu, H., Li, C., Li, Y ., Lee, Y .J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 26296–26306 (June 2024)

    2024

  38. [47]

    In: NeurIPS (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y .J.: Visual instruction tuning. In: NeurIPS (2023)

    2023

  39. [48]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing pp

    Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M.D., Zou, Y ., Wang, W.: WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing pp. 1–15 (2024)

    2024

  40. [49]

    Mou, C., Wang, X., Xie, L., Wu, Y ., Zhang, J., Qi, Z., Shan, Y ., Qie, X.: T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models (2023), https://arxiv.org/abs/ 2302.08453

    2023 arXiv

  41. [50]

    In: ArXiv (2024)

    Ng, E., Romero, J., Bagautdinov, T., Bai, S., Darrell, T., Kanazawa, A., Richard, A.: From audio to photoreal embodiment: Synthesizing humans in conversations. In: ArXiv (2024)

    2024

  42. [51]

    In: CVPR (2016)

    Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E.H., Freeman, W.T.: Visually indicated sounds. In: CVPR (2016)

    2016

  43. [52]

    In: AAAI Conference on Artificial Intelligence (AAAI) (2022)

    Park, S.J., Kim, M., Hong, J., Choi, J., Ro, Y .M.: Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory. In: AAAI Conference on Artificial Intelligence (AAAI) (2022)

    2022

  44. [53]

    Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.Y ., Chuang, C.Y ., Yan, D., Choudhary, D., Wang, D., Sethi, G., Pang, G., Ma, H., Misra, I., Hou, J., Wang, J., Jagadeesh, K., Li, K., Zhang, L., Singh, M., Williamson, M., Le, M., Yu,...

    2025 arXiv

  45. [54]

    In: ICML (2021)

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021)

    2021

  46. [55]

    In: CVPR (2022)

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)

    2022

  47. [56]

    Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core-set approach (2018), https://arxiv.org/abs/1708.00489

    2018 arXiv

  48. [57]

    In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory

    Seung, H.S., Opper, M., Sompolinsky, H.: Query by committee. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory. p. 287–294. COLT ’92, Association for Computing Machin- ery, New York, NY , USA (1992). https://doi.org/10.1145/130385.130417,https://doi...

    1992

  49. [58]

    Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., Taigman, Y .: Make-a-video: Text-to-video generation without text-video data (2022)

    2022

  50. [59]

    org/abs/1703.05175

    Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning (2017), https://arxiv. org/abs/1703.05175

    2017 arXiv

  51. [60]

    Sohn, K., Berthelot, D., Li, C.L., Zhang, Z., Carlini, N., Cubuk, E.D., Kurakin, A., Zhang, H., Raffel, C.: Fixmatch: Simplifying semi-supervised learning with consistency and confidence (2020), https: //arxiv.org/abs/2001.07685

    2020 arXiv

  52. [61]

    In: ICLR (2021)

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: ICLR (2021)

    2021

  53. [62]

    Sung, F., Yang, Y ., Zhang, L., Xiang, T., Torr, P.H.S., Hospedales, T.M.: Learning to compare: Relation network for few-shot learning (2018), https://arxiv.org/abs/1711.06025

    2018 arXiv

  54. [63]

    arXiv preprint arXiv:2411.15098 (2024) 12

    Tan, Z., Liu, S., Yang, X., Xue, Q., Wang, X.: Ominicontrol: Minimal and universal control for diffusion transformer. arXiv preprint arXiv:2411.15098 (2024) 12

    2024 arXiv

  55. [64]

    arXiv preprint arXiv:2503.08280 (2025)

    Tan, Z., Xue, Q., Yang, X., Liu, S., Wang, X.: Ominicontrol2: Efficient conditioning for diffusion transformers. arXiv preprint arXiv:2503.08280 (2025)

    2025 arXiv

  56. [65]

    In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https://openreview

    Tang, Z., Yang, Z., Zhu, C., Zeng, M., Bansal, M.: Any-to-any generation via composable diffusion. In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https://openreview. net/forum?id=2EDqbSCnmF

    2023

  57. [66]

    Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results (2018), https://arxiv.org/abs/1703.01780

    2018 arXiv

  58. [67]

    org/abs/2003.12039

    Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow (2020), https://arxiv. org/abs/2003.12039

    2020 arXiv

  59. [68]

    In: Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019

    Tsuchida, S., Fukayama, S., Hamasaki, M., Goto, M.: Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing. In: Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019. Delft...

    2019

  60. [69]

    In: arXiv (2019)

    Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Towards accurate generative models of video: A new metric & challenges. In: arXiv (2019)

    2019

  61. [70]

    In: NeurIPS (2017)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)

    2017

  62. [71]

    Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning (2017), https://arxiv.org/abs/1606.04080

    2017 arXiv

  63. [72]

    arXiv preprint arXiv:2503.20314 (2025)

    Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., Wang, J., Zhang, J., Zhou, J., Wang, J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Feng, M., Zhang, N., Li, P., Wu, P., Chu, R., Feng, R., Zhang, S., Sun, S., Fang, T., Wang, ...

    2025 arXiv

  64. [73]

    Wang, J., Yuan, H., Chen, D., Zhang, Y ., Wang, X., Zhang, S.: Modelscope text-to-video technical report (2023)

    2023

  65. [74]

    arXiv preprint arXiv:2303.17599 (2023)

    Wang, W., Xie, k., Liu, Z., Chen, H., Cao, Y ., Wang, X., Shen, C.: Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17599 (2023)

    2023 arXiv

  66. [75]

    Wang, X., Liu, J., Wang, Z., Yu, X., Wu, J., Sun, X., Su, Y ., Yuille, A., Liu, Z., Barsoum, E.: Keyvid: Keyframe-aware video diffusion for audio-synchronized visual animation (2025), https://arxiv.org/ abs/2504.09656

    2025

  67. [76]

    Wang, Y ., He, Y ., Li, Y ., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y ., He, C., Luo, P., Liu, Z., Wang, Y ., Wang, L., Qiao, Y .: Internvid: A large-scale video-text dataset for multimodal understanding and generation (2024), https://arxiv.org/abs/2307.06942

    2024 arXiv

  68. [77]

    arXiv preprint arXiv:2310.10769 (2023)

    Wu, R., Chen, L., Yang, T., Guo, C., Li, C., Zhang, X.: Lamp: Learn a motion pattern by few-shot tuning a text-to-image diffusion model. arXiv preprint arXiv:2310.10769 (2023)

    2023 arXiv

  69. [78]

    Xie, Z., Zhang, Z., Cao, Y ., Lin, Y ., Bao, J., Yao, Z., Dai, Q., Hu, H.: Simmim: A simple framework for masked image modeling (2022), https://arxiv.org/abs/2111.09886

    2022 arXiv

  70. [79]

    arXiv preprint arXiv:2310.12190 (2023)

    Xing, J., Xia, M., Zhang, Y ., Chen, H., Yu, W., Liu, H., Wang, X., Wong, T.T., Shan, Y .: Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190 (2023)

    2023 arXiv

  71. [80]

    In: International Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    Xue, H., Hang, T., Zeng, Y ., Sun, Y ., Liu, B., Yang, H., Fu, J., Guo, B.: Advancing high-resolution video- language representation with large-scale video transcriptions. In: International Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    2022

  72. [81]

    Yariv, G., Gat, I., Benaim, S., Wolf, L., Schwartz, I., Adi, Y .: Diverse and aligned audio-to-video generation via text-to-video model adaptation (2023)

    2023

  73. [82]

    In: ICLR (2023)

    Ye, Z., Jiang, Z., Ren, Y ., Liu, J., He, J., Zhao, Z.: Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. In: ICLR (2023)

    2023

  74. [83]

    In: Proceedings of the European Conference on Computer Vision (ECCV) (2024) 13

    Zhang, L., Mo, S., Zhang, Y ., Morgado, P.: Audio-synchronized visual animation. In: Proceedings of the European Conference on Computer Vision (ECCV) (2024) 13

    2024

  75. [84]

    In: IEEE International Conference on Computer Vision (ICCV) (2023)

    Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: IEEE International Conference on Computer Vision (ICCV) (2023)

    2023

  76. [85]

    arXiv preprint arXiv:2310.08465 (2023)

    Zhao, R., Gu, Y ., Wu, J.Z., Zhang, D.J., Liu, J., Wu, W., Keppo, J., Shou, M.Z.: Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08465 (2023)

    2023 arXiv

  77. [86]

    In: AAAI Conference on Artificial Intelligence (AAAI) (2019)

    Zhou, H., Liu, Y ., Liu, Z., Luo, P., Wang, X.: Talking face generation by adversarially disentangled audio-visual representation. In: AAAI Conference on Artificial Intelligence (AAAI) (2019)

    2019

  78. [87]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Zhou, H., Sun, Y ., Wu, W., Loy, C.C., Wang, X., Liu, Z.: Pose-controllable talking face generation by implicitly modularized audio-visual representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    2021

  79. [88]

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models (2023), https://arxiv.org/abs/2304.10592 14

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.