Pith. sign in

REVIEW 2 major objections 5 minor 52 references

Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that using previously generated audio as a negative condition lets a video-to-audio model add complementary sounds step by step, trainable on ordinary single-reference data, improving separability and composite quality.

desk verdict A clever and plausible negative-guidance mechanism for incremental video-to-audio, but the claimed separability gains rest on an untested synthetic-input assumption and test-set-tuned hyperparameters. read the letter →

arxiv 2506.20995 v4 pith:ED63NAPC submitted 2025-06-26 cs.CV cs.LGcs.SDeess.AS

Akio Hayakawa , Masato Ishii , Takashi Shibuya , Yuki Mitsufuji This is my paper · ORCID
classification cs.CVcs.LGcs.SDeess.AS
keywords video-to-audiosynthesisnegativeaudioguidancestep-by-stepgenerationflowmatchingControlNetconceptnegationconditioningFoley
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make video-to-audio synthesis work the way Foley artists actually work: in steps. It claims that a pre-trained flow-matching video-to-audio model can be steered so that each new generation step adds a missing sound event instead of duplicating sounds already produced, using a Negative Audio Guidance term that pushes the generated audio away from the audio condition. The key practical claim is that this capability does not require hard-to-obtain multi-reference training data: the audio-conditioned flow estimator is trained on pairs of non-overlapping audio segments taken from the same video in standard datasets. If the claim holds, creators could layer soundtracks from text prompts, adding or correcting one event at a time, and the paper reports that this improves per-track separability and the quality of the final mixed composite audio.

What carries the argument

The load-bearing object is Negative Audio Guidance (NAG), the guided flow of Eq. (7) that includes the term $\beta(u_\theta(x_t,t,V,C_2)-u_{\theta,\psi}(x_t,t,V,\emptyset,x^{(1)}))$. The first flow comes from the frozen pre-trained video-to-audio model; the second flow is produced by a ControlNet-style adapter trained to predict the velocity direction for audio that is semantically similar to the conditioning audio $x^{(1)}$. Subtracting the two steers the sampler away from configurations that the adapter believes resemble previously generated audio, while the $\alpha$ term keeps the output faithful to the video and the current text prompt. Training the adapter requires only pairs of non-overlapping audio segments from the same video, so standard datasets such as VGGSound and audio-caption corpora suffice.

What would settle it

Run the NAG step on videos whose first track is synthetic and whose second caption names a clearly distinct event, and count how often the second track still contains the first event, by pairwise CLAP similarity, spectrogram inspection, or listener judgment; if a synthetic-mix condition reduces duplication no better than no condition, or substantially worse than a real-audio condition with the same semantic content, the central transfer assumption fails.

Watch

Extended reading notes

Core claim

The central discovery, as the authors state it, is that step-by-step video-to-audio synthesis can be formulated as concept negation: at step $k$, the target distribution for the new track is approximated by $p(x^{(2)} \mid V, C_2, \bar{E}(x^{(1)}))$, where $x^{(1)}$ is the loudness-normalized mix of all tracks generated so far and $\bar{E}(\cdot)$ denotes the negation of the audio condition. The resulting guided flow in Eq. (7), $\tilde{u}_{\theta,\psi}(x_t) = u_\theta(x_t,t,\emptyset,\emptyset) + \alpha(u_\theta(x_t,t,V,C_2)-u_\theta(x_t,t,\emptyset,\emptyset)) + \beta(u_\theta(x_t,t,V,C_2)-u_{\theta,\psi}(x_t,t,V,\emptyset,x^{(1)}))$, combines standard text-and-video classifier-free guidance with a new term that subtracts the prediction of an audio-conditioned flow estimator, pushing generation away from already-present audio. That estimator is implemented as a ControlNet adapter on a frozen MMAudio backbone and trained with an ordinary flow-matching loss on non-overlapping audio-segment pairs from the same video. The authors report that this reduces pairwise inter-track audio similarity, maintains or improves text fidelity and audio-video alignment, and yields composite audio that is preferred over baselines in user tests.

Load-bearing premise

The method assumes that a guidance model trained on real, non-overlapping audio segments from the same video continues to point away from the right content when its conditioning input is a loudness-normalized mix of previously generated synthetic tracks, which the training never directly shows it.

Editorial extensions

If this is right

  • Video-to-audio models can support incremental Foley-style workflows: each step adds a missing event, and the user never needs to regenerate the whole track to fix an omission.
  • Standard single-reference audiovisual datasets are enough to learn the audio-conditioned negative guidance; the paper trains the adapter on non-overlapping segment pairs and uses it on composites of generated tracks.
  • The guidance reduces duplication between tracks: pairwise audio-embedding similarity drops, while text fidelity and video alignment remain essentially at the level of the base model.
  • The composite audio built by mixing five such tracks scores better on automated quality and alignment metrics and is preferred in a user study over composite audio from independent per-track generation.
  • Generation order matters: starting with the most prominent or core sound event gives the best final results, matching how a Foley artist would prioritize.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same training recipe, taking a frozen conditional generator and training a side adapter on pairs of distinct outputs drawn from the same condition, could generalize to other add-to-existing-content tasks such as text-to-music stems or sound-effect libraries.
  • Because the guidance is applied at sampling time, the $\beta$ coefficient is a tunable knob for how aggressively to suppress duplication; the paper's sensitivity analysis shows a separability-quality trade-off that applications could adjust per scene without retraining.
  • A direct stress test for the authors' transfer assumption would be to compare the adapter's behavior when the condition is a real audio segment versus a synthetic mix of generated tracks, which the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a step-by-step video-to-audio (V2A) synthesis method. It trains a ControlNet-augmented MMAudio model to estimate the audio-conditioned flow p(x2|V,E(x1)) from pairs of non-overlapping real audio segments of the same video, and at inference subtracts that flow from the video-text flow in Eq. (7), yielding Negative Audio Guidance (NAG). Five audio tracks per video are generated in a chosen order, loudness-normalized into a composite, and compared against independent generation and negative-prompting baselines on a newly constructed Multi-Caps VGGSound test set. The reported results show improved per-track separability and improved composite audio metrics.

Significance. If validated, the method would be practically useful: it enables incremental sound layering without multi-reference training data, directly addressing an important gap in current V2A systems. The derivation from Eq. (5) to Eq. (7) is internally consistent, the training strategy using paired non-overlapping segments is well motivated, and the evaluation covers both composite audio and per-track quality. The paper also includes a user study and a transparent sensitivity analysis. However, the empirical support is weakened by two load-bearing concerns: hyperparameters and generation order are selected on the same test set used for final reporting, and the audio-conditioned model is trained on real segments but applied to synthetic composite conditions without measuring the distribution shift. These concerns are addressable with additional experiments, but they must be resolved before the main claims can be accepted.

major comments (2)
  1. [§5.1, Appendices E and F] The reported main results are affected by selection on the evaluation set. Appendix E states that α and β were varied and that "we selected α=4.5 and β=1.5 as our default setting" after computing the same metrics reported in Tables 1 and 2 on Multi-Caps VGGSound, and Appendix F selects the descending generation order because it "provides the best results for all metrics" on that same set. Since Tables 1 and 2 are then reported for these selected settings, the headline gains may be inflated by selection bias. Please add a held-out validation split for hyperparameter and order selection, or report the full selection process and final evaluation on a disjoint set.
  2. [§4.2, §5.1, Eq. (7), Appendix G] The audio-conditioned estimator u_{θ,ψ} is trained on pairs of real non-overlapping segments from the same clip, but at inference the conditioning audio x^(1) is a loudness-normalized sum of previously generated synthetic tracks. The paper does not measure how u_{θ,ψ} behaves on this shifted input distribution; if it is miscalibrated, the β-term in Eq. (7) can subtract the wrong content, and the CLAP A-A improvement could partly reflect silence or muffled audio, a failure mode acknowledged in Appendix G. Please add quantitative diagnostics, such as silence/low-energy ratios, a distribution-distance measure between real and synthetic conditioning inputs, or an ablation that conditions on real versus synthetic audio at inference.
minor comments (5)
  1. [Appendix A, Appendix E] The references to "Eq. (9)" are incorrect; the guided flow is numbered Eq. (7) in the main text. Please correct these cross-references.
  2. [§5.1, Appendix C] The captions in Multi-Caps VGGSound are generated by Qwen2.5-VL from video only and are not human-verified; a small human validation study or a caveat in the main text would strengthen the reliability of the text-fidelity comparisons.
  3. [Appendix A, Table A1] The text states that NAG received "significantly higher" ratings for separability, but the table reports only means and standard deviations; please include a significance test or confidence intervals.
  4. [General] Please add a reproducibility statement indicating whether code, model weights, and the Multi-Caps VGGSound captions will be released.
  5. [Table 2] The model name "MMAudio-S-16K" is written with a capital K, which is inconsistent with "MMAudio-S-16k" elsewhere in the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: Eq. (7) follows algebraically from the explicit concept-negation assumption in Eq. (5), and the reported gains are evaluated against external baselines and metrics.

full rationale

The derivation chain is self-contained under the paper's stated assumption. Eq. (5) is an explicit assumption imported from Du et al. (2020), not a conclusion derived in this paper; Eq. (6) follows by Bayes' theorem, and Eq. (7) is obtained by the same coefficient-cancellation simplification used in prior work (Kushwaha and Tian, 2025). No fitted parameter is renamed as a prediction: the ControlNet parameters are trained to estimate the flow for p(x|V,E(x_cond)) and then used as a negative condition in the generative process, rather than being regressed onto the reported evaluation metrics. The main comparison (Ours vs. MMAudio-S-16k, negative prompting, and other open-source baselines) uses external metrics (FD, KL, IS, IB, DeSync, CLAP) and a user study. The self-citation to MMAudio is real prior work and is also used as a baseline; it is not invoked as an unverifiable uniqueness or equivalence result, so it does not constitute load-bearing circularity. Two non-circular concerns are worth noting: (1) the ControlNet is trained on real non-overlapping segments while inference conditions on loudness-normalized mixes of synthetic tracks, a domain shift that could degrade the negation estimate; the paper's Appendix G concedes outputs sometimes suffer from low quality such as silence or muffled sound. (2) The guidance coefficients alpha=4.5 and beta=1.5 are selected on the same test set via the sensitivity study in Appendix E, which may inflate the reported numbers; however, the separability gain is monotonic for all positive beta and the user study is independent, so the central claim does not reduce to the fitted coefficient. No circular step of the enumerated kinds is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a standard flow-matching backend, one borrowed but untested concept-negation assumption, and two empirical data choices (same-video segment pairs and test-set-tuned guidance weights). No new physical entities are introduced; NAG and Multi-Caps VGGSound are method and data artifacts, not postulates.

free parameters (3)
  • alpha (CFG weight) = 4.5
    Coefficient of the standard text/video guidance term in Eq. (7); taken from MMAudio's default and confirmed by sensitivity analysis in Appendix E.
  • beta (NAG weight) = 1.5
    Coefficient of the negative audio guidance term; selected from {0.0, 1.0, 1.5, 2.0} using the same Multi-Caps VGGSound evaluation metrics, i.e., tuned on the test set.
  • Generation order = descending by ImageBind text-video similarity
    Appendix F shows descending order gives the best composite metrics, and this order is used in the main results; another choice made on the same test data.
assumptions (5)
  • standard math Flow-matching and classifier-free guidance formulation (Eq. 1-3)
    Section 3.1; standard generative modeling background used to define the network predictions.
  • domain assumption Concept negation factorization p(x,c_p,not c_n) proportional to p(x)p(c_p|x)p(c_n|x)^-1
    Eq. (5), adopted from Du et al. (2020) and assumed to hold for audio concepts without validation; it is the mathematical basis of the NAG subtraction.
  • standard math Bayes decomposition with conditional independence of C2 and E(x1) given x2 and V
    Appendix B; required for Eq. (A2)-(A4), and assumes the conditioning factors separate as written.
  • domain assumption Non-overlapping same-video audio segments are useful positive pairs for audio conditioning
    Section 4.2; training uses xcond and xtgt from the same video but does not guarantee they contain related sound events, yet the adapter is expected to learn audio context.
  • ad hoc to paper Cancellation of u_theta(V,empty) by setting guidance coefficients w1'=alpha, w3'=beta, w2'=alpha+beta
    Section 4.1 and Appendix B; justified only by the empirical result of Kushwaha and Tian (2025), not by a dedicated error analysis for this audio setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance." pith.science (2026). https://pith.science/paper/ED63NAPC

@misc{pith2026250620995,
  author       = {Pith},
  title        = {Pith review of: Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ED63NAPC}},
  note         = {Machine review of arXiv:2506.20995}
}
read the original abstract

We propose a step-by-step video-to-audio (V2A) generation method that provides finer control over the generation process and more realistic audio synthesis. Inspired by traditional Foley workflows, our approach enables incremental generation of complementary sounds, allowing users to author multiple sound events induced by a video. To avoid the need for costly multi-reference video-audio datasets, each generation step is formulated as a negatively guided V2A process that discourages duplication of sounds already present in previously generated tracks. The guidance model is trained by finetuning a pre-trained V2A model on audio pairs from non-overlapping segments of the same video, encouraging it to leverage acoustic context while remaining visually grounded, and enabling training with standard single-reference audiovisual datasets. Objective and subjective evaluations demonstrate that our method enhances the separability of generated sounds at each step and improves the overall quality of the final composite audio, outperforming existing baselines. Our project page is available at: https://ahykw.github.io/sbsv2a/.

Figures

Figures reproduced from arXiv: 2506.20995 by the authors.

Figure 1
Figure 1. Step-by-step video-to-audio generation for compositional sound effect creation. Video often [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed method. Each audio track should represent a distinct audio event. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of network architecture for the audio-conditional flow estimator. We adopt [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Spectrogram visualizations of step-by-step audio generation using (a) vanilla MMAudio, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Results of user preference comparison between baseline (MMAudio-S-16k) and our method [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 48 canonical work pages

  1. [1]

    The Foley Grail: The Art of Performing Sound for Film, Games, and Animation

    Vanessa Theme Ament. The Foley Grail: The Art of Performing Sound for Film, Games, and Animation. Routledge, 2021

    work page 2021

  2. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  3. [3]

    Erasedraw: Learning to insert objects by erasing them from images

    Alper Canberk, Maksym Bondarenko, Ege Ozguroglu, Ruoshi Liu, and Carl Vondrick. Erasedraw: Learning to insert objects by erasing them from images. In Proceedings of the European Conference on Computer Vision, 2024

    work page 2024

  4. [4]

    Action2sound: Ambient-aware generation of action sounds from egocentric videos

    Changan Chen, Puyuan Peng, Ami Baid, Zihui Xue, Wei-Ning Hsu, David Harwath, and Kristen Grauman. Action2sound: Ambient-aware generation of action sounds from egocentric videos. In Proceedings of the European Conference on Computer Vision, 2024 a

    work page 2024

  5. [5]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2020 a

    work page 2020

  6. [6]

    Generating visually aligned sound from videos

    Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 2020 b

    work page 2020

  7. [7]

    Video-guided foley sound generation with multimodal controls

    Ziyang Chen, Prem Seetharaman, Bryan Russell, Oriol Nieto, David Bourgin, Andrew Owens, and Justin Salamon. Video-guided foley sound generation with multimodal controls. arXiv preprint arXiv:2411.17698, 2024 b

    arXiv 2024

  8. [8]

    Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    work page 2025

Show all 52 references
  1. [9]

    Clotho: An audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2020

    2020

  2. [10]

    Compositional visual generation with energy based models

    Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In Proceedings of the Advances in Neural Information Processing Systems, 2020

    2020

  3. [11]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \"u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image syn...

    2024

  4. [12]

    Sketch2sound: Controllable audio generation via time-varying signals and sonic imitations

    Hugo Flores Garc \' a, Oriol Nieto, Justin Salamon, Bryan Pardo, and Prem Seetharaman. Sketch2sound: Controllable audio generation via time-varying signals and sonic imitations. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2025

    2025

  5. [13]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signa...

    2017

  6. [14]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  7. [15]

    Instructme: An instruction guided music edit and remix framework with latent diffusion models

    Bing Han, Junyu Dai, Weituo Hao, Xinyan He, Dong Guo, Jitong Chen, Yuxuan Wang, Yanmin Qian, and Xuchen Song. Instructme: An instruction guided music edit and remix framework with latent diffusion models. In Proceedings of the International Joint Conference on Artificial Intel...

    2024

  8. [16]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Proceedings of the NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021

    2021

  9. [17]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. In Proceedings of the British Machine Vision Conference, 2021

    2021

  10. [18]

    Synchformer: Efficient synchronization from sparse cues

    Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Synchformer: Efficient synchronization from sparse cues. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2024

    2024

  11. [19]

    Audioeditor: A training-free diffusion-based audio editing framework

    Yuhang Jia, Yang Chen, Jinghua Zhao, Shiwan Zhao, Wenjia Zeng, Yong Chen, and Yong Qin. Audioeditor: A training-free diffusion-based audio editing framework. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2025

    2025

  12. [20]

    Simultaneous music separation and generation using multi-track latent diffusion models

    Tornike Karchkhadze, Mohammad Rasool Izadi, and Shlomo Dubnov. Simultaneous music separation and generation using multi-track latent diffusion models. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2025

    2025

  13. [21]

    Analyzing and improving the training dynamics of diffusion models

    Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  14. [22]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019

    2019

  15. [23]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020

    2020

  16. [24]

    Vintage: Joint video and text conditioning for holistic audio generation

    Saksham Singh Kushwaha and Yapeng Tian. Vintage: Joint video and text conditioning for holistic audio generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    2025

  17. [25]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Prin...

    2023

  18. [26]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In Proceedings of The International Conference on Learning Representations, 2023

    2023

  19. [27]

    Compositional visual generation with composable diffusion models

    Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Proceedings of the European Conference on Computer Vision, 2022

    2022

  20. [28]

    Tell what you hear from what you see-video to audio generation through text

    Xiulong Liu, Kun Su, and Eli Shlizerman. Tell what you hear from what you see-video to audio generation through text. In Proceedings of the Advances in Neural Information Processing Systems, 2024

    2024

  21. [29]

    Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. In Proceedings of the Advances in Neural Information Processing Systems, 2023

    2023

  22. [30]

    Multi-source diffusion models for simultaneous music generation and separation

    Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, and Emanuele Rodol \`a . Multi-source diffusion models for simultaneous music generation and separation. In Proceedings of the International Conference on Learning Representations, 2024

    2024

  23. [31]

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Lang...

    2024

  24. [32]

    Audio-visual scene analysis with self-supervised multisensory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision, 2018

    2018

  25. [33]

    Stemgen: A music generation model that listens

    Julian D Parker, Janne Spijkervet, Katerina Kosta, Furkan Yesiler, Boris Kuznetsov, Ju-Chiang Wang, Matt Avent, Jitong Chen, and Duc Le. Stemgen: A music generation model that listens. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2024

    2024

  26. [34]

    Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Lu...

    arXiv

  27. [35]

    Generalized multi-source inference for text conditioned music diffusion models

    Emilian Postolache, Giorgio Mariani, Luca Cosmo, Emmanouil Benetos, and Emanuele Rodol \`a . Generalized multi-source inference for text conditioned music diffusion models. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2024

    2024

  28. [36]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the Advances in Neural Information Processing Systems, 2016

    2016

  29. [37]

    Smartmask: context aware high-fidelity mask generation for fine-grained object insertion and layout control

    Jaskirat Singh, Jianming Zhang, Qing Liu, Cameron Smith, Zhe Lin, and Liang Zheng. Smartmask: context aware high-fidelity mask generation for fine-grained object insertion and layout control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  30. [38]

    Visually guided sound source separation with audio-visual predictive coding

    Zengjie Song and Zhaoxiang Zhang. Visually guided sound source separation with audio-visual predictive coding. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  31. [39]

    sd3.5, 2024

    Stability-AI. sd3.5, 2024. URL https://github.com/Stability-AI/sd3.5

    2024

  32. [40]

    Steinmetz and Joshua D

    Christian J. Steinmetz and Joshua D. Reiss. pyloudnorm: A simple yet flexible loudness meter in python. In Proceedings of the Audio Engineering Society Convention, 2021

    2021

  33. [41]

    Add-it: Training-free object insertion in images with pretrained diffusion models

    Yoad Tewel, Rinon Gal, Dvir Samuel, Yuval Atzmon, Lior Wolf, and Gal Chechik. Add-it: Training-free object insertion in images with pretrained diffusion models. In Proceedings of the International Conference on Learning Representations, 2025

    2025

  34. [42]

    Liu, Kevin J

    Rafael Valle, Rohan Badlani, Zhifeng Kong, Sang gil Lee, Arushi Goel, Sungwon Kim, Joao Felipe Santos, Shuqi Dai, Siddharth Gururani, Aya Aljafari, Alexander H. Liu, Kevin J. Shih, Ryan Prenger, Wei Ping, Chao-Han Huck Yang, and Bryan Catanzaro. Fugatto 1: Foundational generat...

    2025

  35. [43]

    Temporally aligned audio for video with autoregression

    Ilpo Viertola, Vladimir Iashin, and Esa Rahtu. Temporally aligned audio for video with autoregression. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2025

    2025

  36. [44]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024 a

    2024

  37. [45]

    Frieren: Efficient video-to-audio generation network with rectified flow matching

    Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audio generation network with rectified flow matching. In Proceedings of the Advances in Neural Information Processing Systems, 2024 b

    2024

  38. [46]

    Audit: Audio editing by following instructions with latent diffusion models

    Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, et al. Audit: Audio editing by following instructions with latent diffusion models. In Proceedings of the Advances in Neural Information Processing Systems, 2023

    2023

  39. [47]

    Stable diffusion 2.0 and the importance of negative prompts for good results, 2022

    Max Woolf. Stable diffusion 2.0 and the importance of negative prompts for good results, 2022. URL https://minimaxir.com/2022/11/stable-diffusion-negative-prompt/

    2022

  40. [48]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Si...

    2023

  41. [49]

    Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  42. [50]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    2023

  43. [51]

    Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494, 2024

    2024 arXiv

  44. [52]

    Visually guided sound source separation using cascaded opponent filter network

    Lingyu Zhu and Esa Rahtu. Visually guided sound source separation using cascaded opponent filter network. In Proceedings of the Asian Conference on Computer Vision, 2020

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.