Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diffusion activations are temporally precise enough to replace dedicated feature extractors like CLIP, CLAP, and ImageBind for cross-modal generation, and a symmetric framework built on that idea handles both video-to-audio and…

desk verdict Genuinely new and empirically strong unified V2A/A2V framework using frozen diffusion activations; the reported gains are impressive, but the RoPE alignment mechanism is underspecified and the evaluation has some mild selection issues. read the letter →

arxiv 2412.15191 v2 pith:46AFKBCP submitted 2024-12-19 cs.CV cs.LGcs.SDeess.AS

Moayed Haji-Ali , Willi Menapace , Aliaksandr Siarohin , Ivan Skorokhodov , Alper Canberk , Kwot Sin Lee , Vicente Ordonez , Sergey Tulyakov This is my paper · ORCID
classification cs.CVcs.LGcs.SDeess.AS
keywords video-to-audiogenerationaudio-to-videodiffusionactivationstemporalalignmentflowmatchingfusionblocksfrozengeneratorsrotarypositionembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AV-Link claims that the activations of frozen audio and video diffusion generators carry precise temporal information, enough to replace specialized pretrained feature extractors such as CLIP, CLAP, ImageBind, BEATs, and CAVP in cross-modal generation. It delivers this as a single symmetric framework that handles both video-to-audio and audio-to-video: trainable Fusion Blocks inserted between two frozen flow-matching generators let each modality condition the other through temporally-aligned self-attention. On the VGGSounds benchmark the method raises Onset ACC to 0.531 from the previous best 0.301, and in user studies its temporal alignment is preferred over Movie Gen Audio, a 13B-parameter model, 63.6% of the time. The paper's bet is that diffusion activations, because they are trained to produce temporally-structured output, are better raw material for audio-video synchronization than embeddings learned from contrastive tasks.

What carries the argument

The load-bearing object is the Fusion Block: a multi-head self-attention layer applied to the concatenation of projected audio and video activations, followed by an MLP, whose outputs are reinjected into subsequent DiT blocks of both frozen backbones (symmetric feature reinjection). Temporal correspondence inside the block is set by a time-aligned 1D Rotary Position Embedding, which rotates a token at temporal index n by an angle proportional to n·θ_base for video tokens and n·(η_v/η_a)·θ_base for audio tokens, so that tokens representing the same moment in time share the same rotation. A second design choice matters nearly as much: the conditioning modality is fed to the fusion blocks at a fixed flow timestep near full denoising (0.96 for video-to-audio, 0.8 for audio-to-video), where the high-frequency details that determine sound onsets survive.

What would settle it

Construct a test set in which the same visual event is paired with audio whose onset leads or lags the visual by controlled offsets (for example 50 to 300 milliseconds, varying per clip) and measure whether AV-Link's generated audio tracks the true onsets or locks onto the linear token-ratio grid: the claimed mechanism predicts Onset ACC should collapse as the offset leaves the fixed τ(n) = n·η_v/η_a mapping. A second check: encode the same video content at different frame rates or token densities and see whether alignment degrades, which would show how load-bearing the fixed ratio is.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in the introduction, is that video and audio diffusion activations contain rich temporal information and can for the first time replace specialized pretrained feature extractors while improving temporal alignment. The evidence is an architecture that keeps two pretrained generators frozen and adds only 186M trainable parameters in eight Fusion Blocks, yet surpasses dedicated V2A and A2V systems on quality and alignment metrics while being preferred by human raters on temporal alignment. The conceptual discovery is that the conditioning signal does not have to be a static embedding from a task-specific encoder; it can be the evolving internal state of a generative model of the other modality, refined throughout sampling.

Load-bearing premise

Alignment depends on a single fixed linear mapping between audio time and video time set by the ratio of tokens per second, so any sound that leads or lags its visual event, or any clip sampled at a different rate, falls outside what the mechanism can represent.

Editorial extensions

If this is right

  • Video-to-audio and audio-to-video collapse into one self-contained system: 8 Fusion Blocks (186M parameters) on top of frozen 576M-parameter generators replace the separate feature extractors and task-specific models prior work required.
  • Temporal alignment improves even against far larger systems: human raters prefer AV-Link's timing over Movie Gen Audio (13B parameters) 63.6% of the time, and Onset ACC rises to 0.531 from the 0.301 best prior baseline.
  • A single shared set of Fusion Block parameters can serve both directions with only a marginal drop, so one checkpoint can cover text-to-audio, text-to-video, V2A, and A2V generation.
  • Because both backbones stay frozen, future improvements to single-modality generators should transfer directly to cross-modal quality without retraining the fusion machinery.
  • The conditioning stream should be injected at a high, fixed flow timestep rather than sampled uniformly, since high-frequency visual details drive onset-sensitive sounds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same frozen-generator fusion recipe suggests itself for other timing-critical modality pairs, such as music-to-dance or lip-synchronized speech, wherever each side already has a generator whose activations encode temporal structure; the paper does not test these cases.
  • A stress test the paper does not run: offset sound events from their visuals by varying lead or lag times, or re-encode clips at different frame rates; the fixed token-ratio mapping τ(n) = n·η_v/η_a predicts alignment should break as offsets grow.
  • The finding that near-clean conditioning activations work best is a transferable design principle for conditional generation: for onset-sensitive outputs, destroying high-frequency detail in the conditioning stream is the wrong default, even though uniform noisy-condition sampling is a common training choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes AV-Link, a unified framework for video-to-audio (V2A) and audio-to-video (A2V) generation built from two frozen flow-matching generators. The key components are Fusion Blocks that run self-attention over concatenated audio and video token sequences, a time-aligned 1D RoPE scheme intended to align tokens by seconds, and a symmetric feature reinjection mechanism that feeds cross-modal activations back into the conditioning backbone. The authors claim that diffusion activations contain rich temporal information and can replace specialized feature extractors, and report large gains in Onset ACC on VGGSounds (0.531 vs. 0.301 for the best baseline), competitive semantic metrics, and user-study preferences for temporal alignment over Movie Gen Audio. The paper also provides ablations of the conditioning timestep, fusion block placement, feature types, and feature injection methods.

Significance. If the central claim holds, AV-Link is a useful step toward unified audiovisual generation, since it avoids task-specific encoders (CLIP, CLAP, CAVP, BEATs) and lets two frozen unimodal generators exchange temporally aligned features. The paper's strengths include a broad ablation matrix, quantitative and user evaluations on two benchmarks, reproduction of several baselines from released code, and qualitative in-the-wild tests. The symmetric design and the idea of using diffusion activations as cross-modal conditioning are likely to be of interest to the community. However, the significance is contingent on resolving the test-set tuning of the conditioning timestep, on demonstrating that the temporal alignment mechanism is correctly specified and robust, and on strengthening the A2V comparison.

major comments (4)
  1. [Sec. 3.3, Eq. (4); Sec. 3.2; Appx. B] The temporal alignment mapping in Eq. (4) is underspecified. The paper never states the numerical values of η_a and η_v, and the definition of T_v in §3.2 as 'the total number of video pixels' is ambiguous about whether the video token sequence is ordered temporally or spatiotemporally after the 2×2 patchification described in Appx. B. If tokens are ordered as frame·S + spatial, then tokens from the same video time receive different rotations, so no single audio token can align to a given video time, and the self-attention in the Fusion Block would be smeared over spatial indices. If tokens are ordered purely temporally, the meaning of η_v is unclear because the model uses 3D-RoPE. The authors should specify the exact token layout, give concrete η_a and η_v for the 5.16s/6fps/36×64 setting, and report a diagnostic (e.g., attention localization or a synthetic offset test) showing that the RoPE-based correspondence is what drives the Onset ACC improvement.
  2. [Sec. 4.4, Fig. 3; Sec. 4.2; Tab. 2] The conditioning flow timestep hyperparameters (0.96 for V2A, 0.8 for A2V) are selected by scanning on the VGGSounds test set and then used to train the final models whose results are reported on the same test set. Section 4.4 states that after identifying the optimal timestep, 'we train subsequent models using these fixed timesteps,' and the evaluation protocol in §4.2 uses 2048 videos from the VGGSounds test set. This is a form of test-set tuning that can inflate the reported gains in Tab. 2 and Tab. 1. Please select timesteps on a held-out validation split, or provide evidence (e.g., via the Movie Gen benchmark or a different dataset) that the optimal timesteps transfer across distributions.
  3. [Sec. 4.4; Tabs. 1–3] Automatic metrics are reported without error bars or multiple seeds, and Section 4.4 explicitly states that 'we perform evaluation using a single fixed seed for all test videos.' Several ablation differences in Tab. 2 are small (e.g., Onset ACC 0.415 vs. 0.410 for symmetric cross attention vs. fusion blocks; IB-AV 0.155 vs. 0.150), so with one seed these may not be significant. The central claim that the Fusion Block design improves temporal alignment would be substantially strengthened by reporting means and standard deviations over at least three seeds, and by statistical comparisons for the key ablations.
  4. [Sec. 4.3, Tab. 3; Appx. D.2] The A2V evaluation compares only against TempoTokens, and the comparison crops AV-Link's 5.16s outputs to 2s to match TempoTokens. A single baseline, especially one that generates only 2s clips, is a weak basis for the claim that AV-Link surpasses prior work in A2V; other A2V methods (AADiff, Seeing-and-Hearing A2V, AVSyncD) are discussed but not included in quantitative comparisons. The cropping protocol may also affect FVD and temporal metrics differently for the two methods. Please add at least one additional strong A2V baseline using published numbers where code is unavailable, and analyze the sensitivity of the metrics to the cropping protocol.
minor comments (4)
  1. [Abstract] The abstract writes 'Video-to-Audio (A2V) and Audio-to-Video (A2V)'; the first expansion should be V2A. The running header and Figure 1 caption also contain 'A V-Link' with an extra space.
  2. [Sec. 1] The phrase 'We show for the first time' is stronger than necessary and may invite unnecessary priority disputes; consider softening it, especially given prior work on diffusion hyperfeatures for spatial alignment [56].
  3. [Sec. 3.2] The sentence 'Given an RGB video V1, we flatten it to shape R^{T_v×3}' is confusing because T_v is later called 'the total number of video pixels'; please separate the sequence length (number of tokens) from the spatial dimensions.
  4. [Sec. 4.2; Tab. 1] FID is an image metric; specify how it is applied to video (e.g., per-frame) and whether the same frames are used for all methods. In addition, the ✓ marks in Tab. 1 should be defined in the caption to clarify which results are text-conditioned.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AV-Link is an empirical system evaluated on external benchmarks, and its temporal-alignment mechanism is an explicit design choice rather than a result derived from or equivalent to its inputs.

full rationale

The paper's central claim is that frozen audio and video diffusion activations can serve as temporally aligned cross-modal conditioning. This is supported by a concrete architecture (Fusion Blocks with time-aligned RoPE) and by external evaluations on VGGSounds and the Movie Gen benchmark. The temporal alignment rule in Eq. 4, tau(n) = n * eta_v / eta_a for audio tokens, is an explicit design assumption about how audio and video token indices relate; it is not derived from, nor does it reduce to, the reported Onset ACC or user-study numbers. The improvements over Diff-Foley, FoleyCrafter, Frieren, V2A-Mapper, Seeing-and-Hearing, TempoTokens, and Movie Gen are empirical comparisons, not predictions forced by a fitted parameter. The selection of conditioning flow timesteps (0.96 for V2A, 0.8 for A2V) was made by scanning performance curves, which is hyperparameter tuning on the evaluation setup rather than a circular step: the final numbers are not the same quantity as the tuning curve, and the timestep choice does not by construction determine the Onset ACC advantage. Self-citations to Haji-Ali et al. [27] are used to adopt a frozen pretrained audio backbone and its VAE/vocoder, which is normal use of prior work and is not load-bearing for the claimed novelty; no uniqueness theorem is imported, and no ansatz is smuggled in via citation. The limitations section is transparent about the low-resolution video backbone and added compute. Because no load-bearing step reduces by definition to its own inputs and no fitted quantity is renamed as a prediction, the derivation chain is self-contained.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on the empirical premise that frozen diffusion activations are temporally aligned across modalities, plus the specific linear time-correspondence assumption in the RoPE alignment; the conditioning timesteps are tuned on the evaluation benchmarks, and no invented physical entities are introduced.

free parameters (3)
  • Conditioning flow timestep for V2A (video) = 0.96
    Chosen by scanning t on the evaluation benchmarks (Fig. 3); the paper fixes it for all V2A experiments. Tuning on the evaluation set risks optimistic results.
  • Conditioning flow timestep for A2V (audio) = 0.8
    Chosen by the same tuning procedure (Fig. 3, Section 4.4).
  • Logit-normal location shift for fusion training = -1.0
    Shifted from N(0,1) to N(-1,1) to sample more noisy flow steps; selected for faster convergence (Appendix D.1).
assumptions (4)
  • domain assumption Frozen diffusion model activations encode temporally and semantically aligned information for both audio and video modalities.
    Core premise of the method, stated in Sections 1 and 3.3 without independent verification; supported only by the paper's own experiments.
  • ad hoc to paper Audio and video token positions correspond linearly through the ratio of tokens per second (eta_v / eta_a) under 1D RoPE.
    The temporal alignment function tau(n) in Eq. (4) assumes a fixed linear mapping between audio and video tokens; this is a design choice specific to this paper.
  • domain assumption The paired datasets (VGGSounds, temporally-strong AudioSet, internal high-quality set) provide ground-truth audio-video correspondences that are temporally accurate enough to train alignment.
    Training relies on the dataset alignment being reliable; mislabeled or loosely aligned pairs would degrade the fusion block.
  • standard math Flow matching training objective and linear interpolation path (Eqs. 1 and 2) from prior work are valid and applicable.
    Background method; taken from prior literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation." pith.science (2026). https://pith.science/paper/46AFKBCP

@misc{pith2026241215191,
  author       = {Pith},
  title        = {Pith review of: AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46AFKBCP}},
  note         = {Machine review of arXiv:2412.15191}
}
read the original abstract

We propose AV-Link, a unified framework for Video-to-Audio (A2V) and Audio-to-Video (A2V) generation that leverages the activations of frozen video and audio diffusion models for temporally-aligned cross-modal conditioning. The key to our framework is a Fusion Block that facilitates bidirectional information exchange between video and audio diffusion models through temporally-aligned self attention operations. Unlike prior work that uses dedicated models for A2V and V2A tasks and relies on pretrained feature extractors, AV-Link achieves both tasks in a single self-contained framework, directly leveraging features obtained by the complementary modality (i.e. video features to generate audio, or audio features to generate video). Extensive automatic and subjective evaluations demonstrate that our method achieves a substantial improvement in audio-video synchronization, outperforming more expensive baselines such as the MovieGen video-to-audio model.

Figures

Figures reproduced from arXiv: 2412.15191 by the authors.

Figure 1
Figure 1. Compared to current Video-to-Audio and Audio-to-Video methods, AV-Link provides a unified framework for these two tasks. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Design of the proposed Fusion Block connecting the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of Audio-to-Video and Video-to-Audio [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative V2A results. Our model achieved the best temporal alignment, matching closely the “bouncing” and “drumming” [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of A2V generation. Our model gen [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison between different parametrizations for the [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Qualitative V2A results comparing our method to baselines on in-the-wild videos captured by the authors that require precise [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Examples MMdiffusion generated samples using their released checkpoint. We show that their model suffers from severe [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing

    cs.CV 2025-12 conditional novelty 5.0 of 10

    A dual-branch diffusion transformer with joint video-audio self-attention and a keypoint-based mouth-area loss reports top lip-sync and speech metrics on two benchmarks.

Reference graph

Works this paper leans on

105 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vid- styleode: Disentangled video editing via stylegan and neu- ralodes

    Moayed Haji Ali, Andrew Bond, Tolga Birdal, Duygu Cey- lan, Levent Karacan, Erkut Erdem, and Aykut Erdem. Vid- styleode: Disentangled video editing via stylegan and neu- ralodes. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7523–7534, 2023. 1

    2023

  2. [2]

    Label-efficient se- mantic segmentation with diffusion models

    Dmitry Baranchuk, Ivan Rubachev, Andrey V oynov, Valentin Khrulkov, and Artem Babenko. Label-efficient se- mantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021. 8

    arXiv 2021

  3. [3]

    Sonicdiffusion: Audio-driven image generation and editing with pretrained diffusion models

    Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakas ¸, Duygu Ceylan, Erkut Erdem, and Aykut Er- dem. Sonicdiffusion: Audio-driven image generation and editing with pretrained diffusion models. arXiv preprint arXiv:2405.00878, 2024. 2

    arXiv 2024

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  5. [5]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 2

    2023

  6. [6]

    The mtg-jamendo dataset for au- tomatic music tagging

    Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for au- tomatic music tagging. InMachine Learning for Music Dis- covery Workshop, ICML (ICML 2019) , Long Beach, CA, United States, 2019. 6

    2019

  7. [7]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, et al. Video generation models as world simulators. OpenAI Blog, 1:8, 2024. 1, 2

    2024

  8. [8]

    Ac- tion2sound: Ambient-aware generation of action sounds from egocentric videos

    Changan Chen, Puyuan Peng, Ami Baid, Zihui Xue, Wei- Ning Hsu, David Harwath, and Kristen Grauman. Ac- tion2sound: Ambient-aware generation of action sounds from egocentric videos. In European Conference on Com- puter Vision, pages 277–295. Springer, 2024. 2

    2024

Show all 105 references
  1. [9]

    Semantically consistent video-to-audio generation using multimodal language large model, 2024

    Gehui Chen, Guan’an Wang, Xiaowen Huang, and Jitao Sang. Semantically consistent video-to-audio generation using multimodal language large model, 2024. 2

    2024

  2. [10]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. 2, 5, 6, 3

    2020

  3. [11]

    Beats: audio pre-training with acoustic tokeniz- ers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. Beats: audio pre-training with acoustic tokeniz- ers. In ICML, pages 5178–5193, 2023. 2, 6

    2023

  4. [12]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In International Confere...

    2024

  5. [13]

    Unrav- eling instance associations: A closer look for audio-visual segmentation

    Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, and Gustavo Carneiro. Unrav- eling instance associations: A closer look for audio-visual segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), page...

    2024

  6. [15]

    Video-guided foley sound generation with multimodal con- trols

    Ziyang Chen, Prem Seetharaman, Bryan Russell, Oriol Ni- eto, David Bourgin, Andrew Owens, and Justin Salamon. Video-guided foley sound generation with multimodal con- trols. arXiv preprint arXiv:2411.17698, 2024. 2

    2024 arXiv

  7. [16]

    Taming multimodal joint training for high-quality video-to-audio synthesis

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Taming multimodal joint training for high-quality video-to-audio synthesis. arXiv preprint arXiv:2412.15322, 2024. 3

    2024 arXiv

  8. [17]

    Lova: Long-form video-to-audio generation

    Xin Cheng, Xihua Wang, Yihan Wu, Yuyue Wang, and Rui- hua Song. Lova: Long-form video-to-audio generation. arXiv preprint arXiv:2409.15157, 2024. 2

    2024 arXiv

  9. [18]

    Visionllama: A unified llama backbone for vision tasks

    Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. Visionllama: A unified llama backbone for vision tasks. In European Conference on Computer Vision, pages 1–18. Springer, 2024. 4

    2024

  10. [19]

    FMA: A dataset for music analysis

    Micha ¨el Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. FMA: A dataset for music analysis. In 18th International Society for Music Information Retrieval Conference (ISMIR), 2017. 6

    2017

  11. [20]

    Conditional generation of audio from 9 video via foley analogies

    Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. Conditional generation of audio from 9 video via foley analogies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2426–2436, 2023. 2, 7

    2023

  12. [21]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Is- mail, and Huaming Wang. Clap learning audio concepts from natural language supervision. InICASSP. IEEE, 2023. 1, 3

    2023

  13. [22]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first international conference on mach...

  14. [23]

    Hawley, and Jordi Pons

    Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. In ICML, 2024. 1, 2

    2024

  15. [24]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessi...

    2017

  16. [25]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023. 2, 3, 6, 7

    2023

  17. [26]

    Gotta hear them all: Sound source aware vision to audio genera- tion

    Wei Guo, Heng Wang, Weidong Cai, and Jianbo Ma. Gotta hear them all: Sound source aware vision to audio genera- tion. arXiv preprint arXiv:2411.15447, 2024. 2

    2024 arXiv

  18. [27]

    Taming data and transformers for audio generation

    Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Guha Balakrishnan, Sergey Tulyakov, and Vicente Or- donez. Taming data and transformers for audio generation. arXiv preprint arXiv:2406.19388, 2024. 1, 2, 3, 4, 6

    2024 arXiv

  19. [28]

    Discriminator-guided cooperative diffu- sion for joint audio and video generation

    Akio Hayakawa, Masato Ishii, Takashi Shibuya, and Yuki Mitsufuji. Discriminator-guided cooperative diffu- sion for joint audio and video generation. arXiv preprint arXiv:2405.17842, 2024. 3, 2

    2024 arXiv

  20. [29]

    Unsupervised semantic correspondence using stable diffusion, 2023

    Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hos- sam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion, 2023. 2

    2023

  21. [30]

    Gans trained by a two time-scale update rule converge to a local nash equi- librium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equi- librium. In NIPS, 2017. 6

    2017

  22. [31]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2022. 1

    2021

  23. [32]

    Make-an-audio 2: Temporal-enhanced text-to-audio generation

    Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474,

    arXiv

  24. [33]

    Rhythmic foley: A framework for seamless audio-visual alignment in video-to-audio synthe- sis

    Zhiqi Huang, Dan Luo, Jun Wang, Huan Liao, Zhiheng Li, and Zhiyong Wu. Rhythmic foley: A framework for seamless audio-visual alignment in video-to-audio synthe- sis. arXiv preprint arXiv:2409.08628, 2024. 3

    2024 arXiv

  25. [34]

    Epic-sounds: A large- scale dataset of actions that sound

    Jaesung Huh, Jacob Chalk, Evangelos Kazakos, Dima Damen, and Andrew Zisserman. Epic-sounds: A large- scale dataset of actions that sound. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023. 3

    2023

  26. [35]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. In British Machine Vision Conference (BMVC), 2021. 3

    2021

  27. [36]

    A simple but strong baseline for sounding video generation: Effective adaptation of audio and video diffusion models for joint generation

    Masato Ishii, Akio Hayakawa, Takashi Shibuya, and Yuki Mitsufuji. A simple but strong baseline for sounding video generation: Effective adaptation of audio and video diffusion models for joint generation. arXiv preprint arXiv:2409.17550, 2024. 2, 3

    2024 arXiv

  28. [37]

    The power of sound (tpos): Audio reactive video generation with stable diffusion

    Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Seo, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos): Audio reactive video generation with stable diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7822–7832, 2023. 3

    2023

  29. [38]

    Pyramidal flow matching for efficient video generative modeling

    Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954 ,

  30. [39]

    Re- purposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Re- purposing diffusion-based image generators for monocular depth estimation. In CVPR, 2024. 2

    2024

  31. [40]

    A versatile diffusion trans- former with mixture of noise levels for audiovisual genera- tion

    Gwanghyun Kim, Alonso Martinez, Yu-Chuan Su, Brendan Jou, Jos´e Lezama, Agrim Gupta, Lijun Yu, Lu Jiang, Aren Jansen, Jacob Walker, et al. A versatile diffusion trans- former with mixture of noise levels for audiovisual genera- tion. arXiv preprint arXiv:2405.13762, 2024. 2, 3

    2024 arXiv

  32. [41]

    Hifi-gan: Generative adversarial networks for efficient and high fi- delity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fi- delity speech synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 4

    2020

  33. [42]

    Vintage: Joint video and text conditioning for holistic audio generation

    Saksham Singh Kushwaha and Yapeng Tian. Vintage: Joint video and text conditioning for holistic audio generation. arXiv preprint arXiv:2412.10768, 2024. 2

    2024 arXiv

  34. [43]

    Mandel, Mert Bay, and J

    Edith Law, Kris West, Michael I. Mandel, Mert Bay, and J. S. Downie. Evaluation of algorithms using games: The case of music tagging. In International Society for Music Information Retrieval Conference, 2009. 6

    2009

  35. [44]

    Video-foley: Two-stage video-to-sound generation via tem- poral event condition for foley sound

    Junwon Lee, Jaekwon Im, Dabin Kim, and Juhan Nam. Video-foley: Two-stage video-to-sound generation via tem- poral event condition for foley sound. arXiv preprint arXiv:2408.11915, 2024. 2

    2024

  36. [45]

    Aadiff: Audio-aligned video synthesis with text-to- image diffusion

    Seungwoo Lee, Chaerin Kong, Donghyeon Jeon, and Nojun Kwak. Aadiff: Audio-aligned video synthesis with text-to- image diffusion. arXiv preprint arXiv:2305.04001 , 2023. 1, 3

    2023 arXiv

  37. [46]

    Sound-guided semantic video generation

    Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chany- oung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic video generation. In European Conference on Computer Vision, pages 34–50. Springer, 2022. 2, 3 10

    2022

  38. [47]

    T2v-turbo-v2: Enhancing video generation model post- training through data, reward, and conditional guidance de- sign

    Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robin- son Piramuthu, Wenhu Chen, and William Yang Wang. T2v-turbo-v2: Enhancing video generation model post- training through data, reward, and conditional guidance de- sign. arXiv preprint arXiv:2410.05677, 2024. 2

  39. [48]

    Ross, and Angjoo Kanazawa

    Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa. Learn to dance with aist++: Music conditioned 3d dance generation. arXiv, 2021. 3

    2021

  40. [49]

    Muvi: Video-to-music gener- ation with semantic alignment and rhythmic synchroniza- tion

    Ruiqi Li, Siqi Zheng, Xize Cheng, Ziang Zhang, Sheng- peng Ji, and Zhou Zhao. Muvi: Video-to-music gener- ation with semantic alignment and rhythmic synchroniza- tion. arXiv preprint arXiv:2410.12957, 2024. 2

    2024 arXiv

  41. [50]

    Language-guided joint audio-visual editing via one-shot adaptation

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Language-guided joint audio-visual editing via one-shot adaptation. InProceedings of the Asian Conference on Computer Vision , pages 1011–1027, 2024. 3, 2

    2024

  42. [51]

    Vmas: Video-to-music generation via se- mantic alignment in web music videos

    Yan-Bo Lin, Yu Tian, Linjie Yang, Gedas Bertasius, and Heng Wang. Vmas: Video-to-music generation via se- mantic alignment in web music videos. arXiv preprint arXiv:2409.07450, 2024. 2

    2024 arXiv

  43. [52]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 3

    2022 arXiv

  44. [53]

    Plumbley

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. ICML, 2023. 1, 6

    2023

  45. [54]

    Plumbley

    Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qi- uqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. AudioLDM 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprint arXiv:2308.05734, 2023. 1, 2, 4

    2023 arXiv

  46. [55]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022. 3, 4

    2022 arXiv

  47. [56]

    Diffusion hyperfeatures: Searching through time and space for semantic correspon- dence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspon- dence. Advances in Neural Information Processing Sys- tems, 36:47500–47510, 2023. 2, 8

    2023

  48. [57]

    Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models. NeurIPS, 36, 2024. 1, 2, 3, 4, 5, 6, 7, 8

    2024

  49. [58]

    Videofusion: Decomposed diffusion models for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. CVPR, 2023. 1

    2023

  50. [59]

    Tango 2: Aligning diffusion-based text-to-audio genera- tions through direct preference optimization

    Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio genera- tions through direct preference optimization. In Proceed- ings of the 32nd ACM International Conference on Multi- media, p...

    2024

  51. [60]

    The song describer dataset: a corpus of audio captions for music-and-language evaluation

    Ilaria Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, Elio Quinton, Gy¨orgy Fazekas, and Juhan Nam. The song describer dataset: a corpus of audio captions for music-and-language evaluation. In...

    2023

  52. [61]

    Tavg- bench: Benchmarking text to audible-video generation

    Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Tavg- bench: Benchmarking text to audible-video generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6607–6616, 2024. 2

    2024

  53. [62]

    Foleygen: Visually-guided audio generation

    Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually-guided audio generation. In 2024 IEEE 34th In- ternational Workshop on Machine Learning for Signal Pro- cessing (MLSP), pages 1–6. IEEE, 2024. 2

    2024

  54. [63]

    Snap video: Scaled spatiotemporal transformers for text-to-video synthesis

    Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In CVPR, 2024. 1, 3

    2024

  55. [64]

    Adelson, and William T

    Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. Vi- sually indicated sounds. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 3

    2016

  56. [65]

    Masked generative video-to-audio transform- ers with enhanced synchronicity

    Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, and Joan Serr `a. Masked generative video-to-audio transform- ers with enhanced synchronicity. In European Conference on Computer Vision, pages 247–264. Springer, 2024. 2

    2024

  57. [66]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4195– 4205, 2023. 2, 4, 5, 1

    2023

  58. [67]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Du- moulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018. 4

    2018

  59. [68]

    Sampson, Shikai Li, Si- mone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petro- 11 vic, and Yuming Du

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjan- dra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Ja- gadeesh, Kunpeng Li...

    2024

  60. [69]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 2, 3, 6

    2021

  61. [70]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2022. 4

    2022

  62. [71]

    Egosonics: Generat- ing synchronized audio for silent egocentric videos

    Aashish Rai and Srinath Sridhar. Egosonics: Generat- ing synchronized audio for silent egocentric videos. arXiv preprint arXiv:2407.20592, 2024. 2

    2024 arXiv

  63. [72]

    Sta-v2a: Video-to-audio genera- tion with semantic and temporal alignment

    Yong Ren, Chenxing Li, Manjie Xu, Wei Liang, Yu Gu, Rilin Chen, and Dong Yu. Sta-v2a: Video-to-audio genera- tion with semantic and temporal alignment. arXiv preprint arXiv:2409.08601, 2024. 3

    2024 arXiv

  64. [73]

    Mm-diffusion: Learning multi-modal diffusion mod- els for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion mod- els for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2023

  65. [74]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016. 6

    2016

  66. [75]

    I hear your true colors: Im- age guided audio generation

    Roy Sheffer and Yossi Adi. I hear your true colors: Im- age guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 3

    2023

  67. [76]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023. 1

    2023

  68. [77]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024. 4, 5, 1

    2024

  69. [78]

    From vision to audio and beyond: A unified model for audio-visual repre- sentation and generation

    Kun Su, Xiulong Liu, and Eli Shlizerman. From vision to audio and beyond: A unified model for audio-visual repre- sentation and generation. arXiv preprint arXiv:2409.19132,

    arXiv

  70. [79]

    Mm-ldm: Multi-modal latent diffusion model for sounding video generation

    Mingzhen Sun, Weining Wang, Yanyuan Qiao, Jiahui Sun, Zihan Qin, Longteng Guo, Xinxin Zhu, and Jing Liu. Mm-ldm: Multi-modal latent diffusion model for sounding video generation. In Proceedings of the 32nd ACM Inter- national Conference on Multimedia , pages 10853–10861,

  71. [80]

    Motion to dance music generation using latent dif- fusion model

    Vanessa Tan, Junghyun Nam, Juhan Nam, and Junyong Noh. Motion to dance music generation using latent dif- fusion model. In SIGGRAPH Asia 2023 Technical Com- munications, New York, NY , USA, 2023. Association for Computing Machinery. 2

    2023

  72. [81]

    Sequential contrastive audio-visual learning

    Ioannis Tsiamas, Santiago Pascual, Chunghsin Yeh, and Joan Serr `a. Sequential contrastive audio-visual learning. arXiv preprint arXiv:2407.05782, 2024. 3, 2

    2024 arXiv

  73. [82]

    Towards accurate generative models of video: A new met- ric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new met- ric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  74. [83]

    Temporally aligned audio for video with autoregression

    Ilpo Viertola, Vladimir Iashin, and Esa Rahtu. Temporally aligned audio for video with autoregression. arXiv preprint arXiv:2409.13689, 2024. 2, 7

    2024 arXiv

  75. [84]

    Phenaki: Variable length video generation from open do- main textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open do- main textual descriptions. In International Conference on ...

    2022

  76. [85]

    Audiobox: Unified audio generation with natural language prompts

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821, 2023. 1

    2023 arXiv

  77. [86]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foun- dation models

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foun- dation models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 15492–15501, 2024. 3, 5, 6, 7

    2024

  78. [87]

    Av-dit: Efficient audio-visual diffusion transformer for joint audio and video generation

    Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, and Yapeng Tian. Av-dit: Efficient audio-visual diffusion transformer for joint audio and video generation. arXiv preprint arXiv:2406.07686, 2024. 2, 3, 8

    2024 arXiv

  79. [88]

    Tiva: Time-aligned video-to-audio generation

    Xihua Wang, Yuyue Wang, Yihan Wu, Ruihua Song, Xu Tan, Zehua Chen, Hongteng Xu, and Guodong Sui. Tiva: Time-aligned video-to-audio generation. In Proceedings of the 32nd ACM International Conference on Multime- dia (MM ’24), page 10 pages, Melbourne, VIC, Australia,

  80. [89]

    Frieren: Efficient video-to-audio generation with rectified flow matching

    Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audio generation with rectified flow matching. arXiv e-prints, pages arXiv–2406,

  81. [90]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023. 6

    2023

  82. [91]

    Son- icvisionlm: Playing sound with vision language models

    Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. Son- icvisionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 26866–26875, 2024. 3

    2024

  83. [92]

    Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners. In CVPR, pages 7151–7161, 2024. 1, 3, 4, 5, 6, 7, 8, 2

    2024

  84. [93]

    Demysti- fying clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, 12 Luke Zettlemoyer, and Christoph Feichtenhofer. Demysti- fying clip data. arXiv preprint arXiv:2309.16671, 2023. 2, 3

    2023 arXiv

  85. [94]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 2955–2966, 2023. 8

    2023

  86. [95]

    Auf- fusion: Leveraging the power of diffusion and large lan- guage models for text-to-audio generation

    Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Auf- fusion: Leveraging the power of diffusion and large lan- guage models for text-to-audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing,

  87. [96]

    Draw an audio: Leveraging multi-instruction for video-to- audio synthesis

    Qi Yang, Binjie Mao, Zili Wang, Xing Nie, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. Draw an audio: Leveraging multi-instruction for video-to- audio synthesis. arXiv preprint arXiv:2409.06135 , 2024. 2

    2024 arXiv

  88. [97]

    Cmmd: Contrastive multi-modal diffusion for video-audio conditional modeling

    Ruihan Yang, Hannes Gamper, and Sebastian Braun. Cmmd: Contrastive multi-modal diffusion for video-audio conditional modeling. arXiv preprint arXiv:2312.05412 ,

    arXiv

  89. [98]

    Diffusion model as rep- resentation learner

    Xingyi Yang and Xinchao Wang. Diffusion model as rep- resentation learner. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 18938– 18949, 2023. 2

    2023

  90. [99]

    Cogvideox: Text-to- video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to- video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 1, 2, 3, 4

    2024 arXiv

  91. [100]

    Diverse and aligned audio-to- video generation via text-to-video model adaptation

    Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to- video generation via text-to-video model adaptation. In AAAI, pages 6639–6647, 2024. 1, 3, 6, 7, 2

    2024

  92. [101]

    Momu-diffusion: On learning long-term motion-music synchronization and cor- respondence

    Fuming You, Minghui Fang, Li Tang, Rongjie Huang, Yongqi Wang, and Zhou Zhao. Momu-diffusion: On learning long-term motion-music synchronization and cor- respondence. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2

    2024

  93. [102]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 4

    arXiv

  94. [103]

    Audio-synchronized visual animation

    Lin Zhang, Shentong Mo, Yijing Zhang, and Pedro Mor- gado. Audio-synchronized visual animation. In European Conference on Computer Vision , pages 1–18. Springer,

  95. [104]

    Foley- crafter: Bring silent videos to life with lifelike and synchro- nized sounds

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foley- crafter: Bring silent videos to life with lifelike and synchro- nized sounds. arXiv preprint arXiv:2407.01494, 2024. 1, 3, 4, 5, 6, 7, 2

    2024 arXiv

  96. [105]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liangchen Luo, Chien chin Huang, Min Xu, Less Wright, Hamid Shojanaz- eri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Bali- oglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded da...

    2023

  97. [106]

    Which audio has the best quality? Only listen to the audio and ignore the video content

    Yuan Zhou, Qiuyue Wang, Yuxuan Cai, and Huan Yang. Allegro: Open the black box of commercial-level video generation model. arXiv preprint arXiv:2410.15458, 2024. 3 13 A V-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation Supplementary Material ...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.