Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation

Jiasheng Lu; Keyu Fan; Peiwu Qin; Qingyang Shi; Xinyu Zhang; Yingshan Liang; Yiran Wang; Zhicheng Du

REVIEW 4 major objections 6 minor 57 references

Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hear-Your-Click makes video-to-audio generation object-specific and click-controlled, reporting the best scores on VGG-AnimSeg.

desk verdict The click-driven object-specific V2A idea is real and worth building on, but the paper's central SOTA claim is currently unverifiable because Tables 1 and 2 report incompatible scores for the same MVE+CLIP model. read the letter →

arxiv 2507.04959 v2 pith:TSFQLRLV submitted 2025-07-07 cs.CV cs.AIcs.MMcs.SDeess.AS

Yingshan Liang , Keyu Fan , Zhicheng Du , Yiran Wang , Qingyang Shi , Xinyu Zhang , Jiasheng Lu , Peiwu Qin This is my paper

classification cs.CVcs.AIcs.MMcs.SDeess.AS

keywords video-to-audiogenerationinteractiveobject-specificaudiomask-guidedvisualencodercontrastiveaudio-visuallearninglatentdiffusionmodelalignmentVGG-AnimSegdataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that video-to-audio generation can be made object-specific by letting a user click on a frame to select the sound source, and that this click-conditioned design beats global video-to-audio models on a curated animal-video dataset. It argues that the key is to extract visual features only from the masked object, align them with audio through contrastive fine-tuning, and feed them to a latent diffusion model. If the central claim is right, content creators could generate a sound effect for a single object in a crowded scene with one click, and the proposed CAV score would provide a quantitative check that generated audio matches the selected visual content.

What carries the argument

The load-bearing object is the Mask-guided Visual Encoder (MVE), a dual-branch encoder that takes the masked video $V \odot M$ through a video branch and the binary masks $M$ through a mask branch, normalizes and sums the two feature streams. OCAV then aligns these object-level visual features with audio features through a symmetric contrastive loss, initialized from Diff-Foley's pretrained encoders. For generation, the conditioning vector is $x_c = x_v + x_v^*$, where $x_v$ comes from MVE and $x_v^*$ is CLIP features of the masked frames, and a latent diffusion model is trained to denoise a spectrogram latent conditioned on $x_c$. Two augmentations support the alignment: Mask-guided Loudness Modulation scales audio amplitude by the normalized unmasked-pixel ratio per frame, and Random Video Stitching concatenates two videos and overlaps their audio to force attention to individual objects.

What would settle it

Run Hear-Your-Click on the identical test clips with random masks that have the same size and motion as the real object masks; if FD, FAD, KL, and KID do not worsen, the object-specific mask is not doing the claimed work. Separately, collect human ratings of whether generated audio matches the clicked object and compare them with CAV scores; if CAV and humans disagree, the metric-based evidence for audio-visual correspondence is not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that replacing global visual context with mask-conditioned object-level visual features is enough to make a video-to-audio diffusion model generate the sound of a clicked object and synchronize loudness with the object's on-screen size and distance. On the VGG-AnimSeg dataset (about 27,000 animal-video clips built from VGGSound by filtering with CLIP/CLAP similarity and segmenting with DEVA), Hear-Your-Click reports the best FD, FAD, KL, KID, and CAV scores among SpecVQGAN, Im2Wav, Seeing and Hearing, and Diff-Foley. The method also claims that the proposed CAV score, computed with a pretrained multimodal contrastive model, quantifies audio-video correspondence better than the existing metrics alone. The user-facing claim is that one click on a frame is enough to specify the target object, with SAM and TAM turning the click into a propagated mask.

Load-bearing premise

The comparison in Table 1 is fair: each baseline runs in its original configuration on the same test videos, so the reported advantage of Hear-Your-Click reflects the mask-conditioned design rather than dataset filtering, sampling protocol, or the unvalidated CAV metric.

Editorial extensions

If this is right

A single user click on one frame, propagated by SAM and TAM into a video mask, is enough to specify which object should produce sound in the generated audio.
The mask-conditioned visual features (MVE) lower FD, FAD, KL, and KID relative to global CLIP or CAVP features, and adding CLIP features to MVE features improves the scores further.
Mask-guided Loudness Modulation makes generated loudness track the target object's on-screen size and distance, while Random Video Stitching improves multi-object scene handling even though it slightly hurts aggregate metrics.
The CAV score gives a numeric handle on audio-video correspondence that ranks Hear-Your-Click above the compared baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mask-conditioned conditioning is the real cause of the gains, the same recipe should transfer to non-animal videos and to other latent diffusion backbones, which the paper does not test.
The CAV score is only validated on this dataset; a straightforward extension would be to check it against human ratings of correspondence on diverse videos, since a metric that agrees with humans would be useful beyond V2A.
Random Video Stitching's small aggregate loss but clear multi-object benefit suggests that an aspect-ratio-preserving version, or a curriculum that applies stitching only to multi-object clips, could retain both improvements.
The method's reliance on automatically generated masks means its upper bound is the quality of the segmentation; testing with hand-refined masks could separate mask quality from generation quality.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The click-driven object-specific video-to-audio idea is real, and it is the part worth keeping. Hear-Your-Click adds a user interaction loop (SAM/TAM clicks) to the V2A pipeline, proposes a mask-guided visual encoder with contrastive fine-tuning, and contributes two augmentations (RVS, MLM) plus a curated animal subset of VGGSound with DEVA masks. The system-level capability—select an object and hear its sound—is genuinely new among the cited V2A baselines, and the authors are honest that RVS hurts FD/KL/KID in the ablation even though it helps multi-object handling qualitatively. That alone puts this above a desk reject.

The problem is that the paper's central SOTA claim is not currently verifiable. Table 1 lists Ours2 (MVE+CLIP) with FD 48.78 and IS 4.49; Table 2, which also tests MVE+CLIP, reports FD 35.41 and IS 5.93 on the same test set. Those are not sampling-noise differences. The paper never states which inference protocol produced Table 1—one sample per video versus ten, dataset masks versus SAM/TAM-propagated masks, which checkpoint. Until that is reconciled, the margin over Diff-Foley could be an artifact of a more favorable evaluation setting rather than the mask-conditioned design.

The other issues are more standard. Baselines are not retrained on VGG-AnimSeg nor given masks, so the comparison is biased even if unintentionally. The new CAV metric is introduced without validation against human judgments, and it uses C-MCR, the same contrastive audio-visual family that the method fine-tunes; that makes part of the evaluation self-referential even though FD/FAD/KL/KID are external. No error bars or significance tests appear anywhere, which matters especially when Table 3 shows RVS moving FD by roughly fifteen points.

None of this refutes the core idea. The architecture is plausible, the qualitative figures suggest real object-sound correspondence, and the authors disclose the RVS trade-off instead of hiding it. But the empirical section as written cannot carry the state-of-the-art claim. A serious referee should engage with it, and the revision path is clear: reconcile Tables 1 and 2, state the inference protocol, retrain or mask-condition the baselines, validate CAV, and add variance estimates. I would not cite the quantitative numbers in the current form, but I would follow the work and would send it to review.

Referee Report

4 major / 6 minor

Summary. The paper proposes Hear-Your-Click, an interactive video-to-audio (V2A) framework in which a user clicks an object in a frame and the system generates audio tailored to that object. The method introduces a VGG-AnimSeg dataset built from VGGSound animal videos with DEVA-generated masks, an Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) objective with a Mask-guided Visual Encoder (MVE), two data augmentations (Random Video Stitching and Mask-guided Loudness Modulation), and a new CAV evaluation score based on C-MCR contrastive audio-visual representations. The authors report state-of-the-art results on their dataset against SpecVQGAN, Im2Wav, Seeing-and-Hearing, and Diff-Foley, and provide an interactive SAM/TAM-based inference pipeline.

Significance. If the empirical claims hold, this is a useful step toward fine-grained, user-controllable V2A generation, which is a timely problem given the recent interest in interactive and object-centric generative models. The paper ships a new dataset, a concrete training recipe, and an interactive demo, and the core idea of conditioning generation on user-provided masks is sensible and likely to be adopted by the community. However, the significance is currently capped by the empirical verification: the main state-of-the-art claim rests on a single comparison table whose protocol is internally inconsistent with the ablation table, and the new CAV metric is neither validated against human perception nor shown to be independent of the very contrastive objective used to train the model.

major comments (4)

[Sec. 4.2.2 (Table 1) vs. Sec. 4.3.1 (Table 2)] The same model configuration, MVE+CLIP, is reported with substantially different scores in the two tables: FD 48.78 vs. 35.41, IS 4.49 vs. 5.93, FAD 5.02 vs. 4.90, KL 2.82 vs. 2.90, KID 0.010 vs. 0.011, and CAV 2.67 vs. 2.69. The 13-point FD gap and the 1.4-point IS gap cannot be explained by sampling noise, so the two tables must use different inference protocols (for example, different numbers of generated samples per video, different mask sources, or different feature extractors). The paper does not state which protocol produced Table 1. This is load-bearing because Table 1 is the only head-to-head evidence for the state-of-the-art claim; until the discrepancy is reconciled, the reported margins over Diff-Foley and other baselines cannot be attributed to the method itself rather than to a more favorable evaluation setting.
[Sec. 4.2.2] The baseline comparison in Table 1 is not fully specified. The baselines (SpecVQGAN, Im2Wav, SAH, Diff-Foley) are described as recently published open-source methods, but the paper does not state whether they were retrained on VGG-AnimSeg, whether they were given access to the masks, or how their checkpoints and inference hyperparameters were selected. If the baselines were run in their original configurations on a new dataset with different statistics, the comparison may be biased against them. The authors should report the exact evaluation protocol for each baseline, including checkpoint source, number of generated samples per video, and any dataset-specific adaptation, so that the state-of-the-art claim is verifiable.
[Sec. 4.1.2 (CAV score)] The CAV score is introduced as a new metric, but it is not validated. It is computed with C-MCR, a contrastive model that integrates CLIP and CLAP embeddings, which is the same family of contrastive audio-visual representations that the proposed OCAV objective optimizes. This makes the metric partly self-referential: the model is trained to align object-level visual features with audio features in a CLIP-like space, and then evaluated by similarity in another CLIP/CLAP space. The authors should provide evidence that CAV correlates with human judgments of audio-visual correspondence, and ideally report how it behaves when the conditioning visual features are mismatched. Without such validation, CAV should be treated as supporting evidence only, not as a headline metric.
[Tables 1-3] No error bars, confidence intervals, or significance tests are reported for any of the quantitative results. Several claims rest on small metric differences (for example, KL 2.82 vs. 2.90 in Tables 1 and 2, and the MLM/RVS ablations in Table 3). The paper should report results over multiple random seeds or inference runs and, where appropriate, compute statistical significance, especially for the RVS ablation that appears to degrade several metrics in Table 3.

minor comments (6)

[Sec. 5 (Conclusion)] The word 'develope' should be 'develop' in the concluding paragraph.
[Sec. 3.2 (VGG-AnimSeg)] The dataset filtering procedure uses CLAP and CLIP similarity thresholds, but the actual threshold values are not reported. Please specify them, along with the distribution of similarity scores, so that the dataset construction is reproducible.
[Sec. 3.3.1 (MVE)] The mask branch fusion in Eq. (3) uses simple feature addition. The paper should justify why a simple sum was chosen over concatenation or attention-based fusion, especially since the mask stream and video stream may have different feature statistics.
[Sec. 3.4 (Latent Diffusion Model)] The paper states that CLIP features of masked frames are added to MVE features, but it does not clarify whether the CLIP features are extracted from the masked video frames or from the original unmasked frames. Please clarify, as this affects the interpretation of the MVE+CLIP condition.
[Sec. 4.1.3 (Implementation Details)] The audio hop size differs between OCAV training (250) and LDM training (256). Please explain why this discrepancy is necessary and whether it affects the temporal alignment between visual features and audio features.
[Figures 3 and 4] Figures 3 and 4 are not explicitly referenced or described in the main text; they appear to illustrate concepts discussed in Sec. 3.3.3 and Sec. 3.3.4, but the reader cannot tell what the figures show. Please add explicit references and explanations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain from mask-conditioned visual features to generated audio is self-contained, and the CAV metric uses an external C-MCR model rather than the paper's own trained encoders.

full rationale

The paper's central pipeline is click -> SAM/TAM mask -> MVE(+CLIP) visual features -> LDM condition -> audio, with OCAV training the MVE via an InfoNCE objective against a separately initialized audio encoder. None of the equations redefines the target as an input: Eq. (5) is a standard contrastive loss on MVE/PANNs features, while the CAV score (Sec. 4.1.2) uses the external C-MCR model integrating CLIP and CLAP, so evaluation does not reuse the paper's own trained representations. The VGG-AnimSeg test set is filtered by CLIP/CLAP similarity during construction, which may inflate absolute CAV scores, but this is a dataset-selection bias affecting all methods, not a derivation that reduces to its own input. Diff-Foley, C-MCR, CLAP, CLIP, SAM, and TAM are all external references with no author overlap, so no self-citation chain is load-bearing. The discrepancy between Tables 1 and 2 for the same MVE+CLIP configuration (FD 48.78 vs 35.41) is a serious reproducibility and protocol-transparency concern, but it is an empirical inconsistency, not a circular definition or fitted-input-as-prediction. Therefore the core claims are not circular, even though the experimental evidence needs clarification.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central empirical claim rests on the reliability of pretrained audio-visual encoders, the accuracy of automatic video segmentation masks, the validity of mask-area as a loudness proxy, and the fairness of the evaluation versus non-mask baselines.

free parameters (4)

contrastive temperature tau
Temperature in the contrastive loss (Eq. 5) is not reported; it controls the sharpness of the alignment and is a hand-set hyperparameter.
CLAP/CLIP similarity selection thresholds
Dataset filtering in Sec 3.2 selects samples by 'highest average similarity' but does not specify the threshold or how the per-class count of 400/40 was chosen.
guidance scales = CFG=4.5, classifier guidance=50
Inference hyperparameters in Sec 4.1.3 are set without an ablation or justification.
RVS augmentation ratio = 6,800 extra samples from 1,126 base samples
The mapping from 1,126 base samples (Sec 3.3.4) to 6,800 augmented samples (Sec 4.1.1) is unexplained, leaving the augmentation schedule ambiguous.

assumptions (5)

domain assumption Pretrained encoders (Diff-Foley SlowOnly, PANNs, CLIP, C-MCR) provide reliable, transferable representations for audio, video, and text.
Used throughout Secs 3.3 and 3.4 for initialization and feature extraction; their robustness and transferability are taken as given.
domain assumption DEVA text-prompted video segmentation produces masks that accurately isolate the sounding object.
Dataset construction in Sec 3.2 relies on DEVA masks to define the object-level supervision signal.
domain assumption The mask-area ratio is a valid proxy for object distance and therefore for audio loudness.
MLM in Sec 3.3.3 scales audio by the normalized mask pixel fraction; this physical assumption is not validated against real distance-loudness relationships.
domain assumption C-MCR's audio-image contrastive similarity reflects human-perceived audio-visual correspondence.
The CAV score in Sec 4.1.2 uses C-MCR embeddings without any human validation or calibration.
standard math The latent diffusion architecture of Stable Diffusion can be repurposed to generate audio spectrograms conditioned on visual features.
Inherited from Diff-Foley in Sec 3.4; the paper relies on this existing methodology.

invented entities (2)

CAV score
purpose: New automatic evaluation metric for audio-visual correspondence.
Defined via C-MCR embeddings in Sec 4.1.2; no human validation or calibration against listeners, so its validity is not independently established.
VGG-AnimSeg dataset
purpose: Filtered animal-video dataset with masks for training and evaluating object-specific V2A.
Constructed from VGGSound in Sec 3.2; no public release or external verification; filtering criteria are internal to the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation." pith.science (2026). https://pith.science/paper/TSFQLRLV

@misc{pith2026250704959,
  author       = {Pith},
  title        = {Pith review of: Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TSFQLRLV}},
  note         = {Machine review of arXiv:2507.04959}
}

read the original abstract

Video-to-audio (V2A) generation shows great potential in fields such as film production. Despite significant advances, current V2A methods relying on global video information struggle with complex scenes and generating audio tailored to specific objects. To address these limitations, we introduce Hear-Your-Click, an interactive V2A framework enabling users to generate sounds for specific objects by clicking on the frame. To achieve this, we propose Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) with a Mask-guided Visual Encoder (MVE) to obtain object-level visual features aligned with audio. Furthermore, we tailor two data augmentation strategies, Random Video Stitching (RVS) and Mask-guided Loudness Modulation (MLM), to enhance the model's sensitivity to segmented objects. To measure audio-visual correspondence, we designed a new evaluation metric, the CAV score. Extensive experiments demonstrate that our framework offers more precise control and improves generation performance across various metrics. Project Page: https://github.com/SynapGrid/Hear-Your-Click

Figures

Figures reproduced from arXiv: 2507.04959 by the authors.

**Figure 1.** Overview of the Hear-Your-Click system. After users select a single frame and interactively choose specific objects or areas, Hear-Your-Click generates the corresponding audio. This user-friendly approach empowers individuals to customize the audio according to their preferences and interests. Abstract Video-to-audio (V2A) generation shows great potential in fields such as film production. Despite significant advanc… view at source ↗

**Figure 2.** OCAV architecture overview, showing the alignment of object-level visual features from MVE with corresponding audio features. Training data augmentation includes random video stitching and mask-guided loudness modulation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Correlation enhancement between object distance and audio volume changes achieved through adaptive filtering [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Overview of interactive inference process. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Visualization of generated audio samples: (a) Reproduction of ground truth sounds with preserved synchronization, demonstrating accurate capture of transient motion. (b) Audio volume reduction corresponding to cows running away, while maintaining finer details. [29]. T…

**Figure 7.** Figure 7: Comparison of audio generation results using different visual features, demonstrating that MVE and MVE+CLIP capture semantic information and temporal context more effectively than other methods, which produce target-agnostic or low-fidelity audio [PITH_FULL_IMAGE:fig…

**Figure 8.** Figure 8: demonstrates that the model trained with MLM generates audio with more pronounced changes in loudness corresponding to the distance of the target object. However, adding RVS to our base model slightly decreases performance. This may be due to the resizing process in…

**Figure 9.** Figure 9: Ablation study of Random Video Stitching (RVS), demonstrating its role in enabling the model to generate distinct sounds corresponding to multiple selected objects in the scene. augmentation. arXiv preprint arXiv:2304.14601 (2023). [6] Benjamin Elizalde, Soham Deshmukh…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 39 canonical work pages

[1]

Gehui Chen, Guan’an Wang, Xiaowen Huang, and Jitao Sang. 2024. Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model. arXiv preprint arXiv:2404.16305 (2024)

arXiv 2024
[2]

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman

work page
[3]

Lihui Chen, Gemine Vivone, Zihao Nie, Jocelyn Chanussot, and Xi- aomin Yang. 2023. Spatial data augmentation: Improving the general- ization of neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing 61 (2023), 1–11

work page 2023
[4]

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. 2023. Tracking anything with decoupled video segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1316–1326

work page 2023
[5]

Jinhao Duan, Quanfu Fan, Hao Cheng, Xiaoshuang Shi, and Kaidi Xu. 2023. Improve video representation with temporal adversarial Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation , , Figure 9. Ablation study of Random Video Stitching (RVS), demonstrating its role in enabling the model to generate distinct sounds corresponding to multipl...

work page Pith review arXiv 2023
[6]

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huam- ing Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 1–5

work page 2023
[7]

Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons

work page
[8]

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He

work page

Show all 57 references

[9]

Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab. 2023. Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 16144–16154

2023
[10]

Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. 2023. Token- flow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373 (2023)

2023 arXiv
[11]

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Im- agebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15180–15190

2023
[12]

Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. 2022. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839 (2022)

2022 arXiv
[13]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139– 144

2020
[14]

Yuxin Guo, Siyang Sun, Shuailei Ma, Kecheng Zheng, Xiaoyi Bao, Shijie Ma, Wei Zou, and Yun Zheng. 2024. CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

2024
[15]

Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 976–980

2022
[16]

Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. 2023. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision . 379–389

2023
[17]

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16000–16009

2022
[18]

Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al. 2024. Mavil: Masked audio-video learners. Advances in Neural Information Processing Systems 36 (2024)

2024
[19]

Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. 2023. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917 (2023)

2023 arXiv
[20]

Vladimir Iashin and Esa Rahtu. 2021. Taming visually guided sound generation. arXiv preprint arXiv:2110.08791 (2021)

2021 arXiv
[21]

Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. 2024. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403.06976 (2024)

2024 arXiv
[22]

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rol- land, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 4015–4026

2023
[23]

Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 28 (2020), 2880–2894

2020
[24]

Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski
[25]

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. 2023. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 (2023)

2023 arXiv
[26]

Plumbley

Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley
[27]

Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, An- shumali Shrivastava, and Andrew Gordon Wilson. 2022. Learn- ing multimodal data augmentation in feature space. arXiv preprint arXiv:2212.14453 (2022)

2022 arXiv
[28]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Generative image dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 24142–24153
[29]

Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. 2024. Diff- foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems 36 (2024)

2024
[30]

Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung-Yeung Shum, Wei Liu, et al
[31]

IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (2024), 2871–2883

AudioLDM 2: Learning Holistic Audio Generation With Self- Supervised Pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (2024), 2871–2883. https://doi.org/10.1109/ TASLP.2024.3399607

2024
[32]

Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, and Joan Serrà
[33]

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 (2022)

2022 arXiv
[34]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695

2022
[35]

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubin- stein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 22...

2023
[36]

arXiv preprint arXiv:2403.08268 (2024)

Follow-your-click: Open-domain regional image animation via short prompts. arXiv preprint arXiv:2403.08268 (2024). , , Liang et al

2024 arXiv
[37]

Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. 2021. Robust audio-visual instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 12934–12945

2021
[38]

Ioannis Tsiamas, Santiago Pascual, Chunghsin Yeh, and Joan Serrà
[39]

arXiv preprint arXiv:2407.10387 (2024)

Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity. arXiv preprint arXiv:2407.10387 (2024)

2024 arXiv
[40]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

2021
[41]

Xihua Wang, Yuyue Wang, Yihan Wu, Ruihua Song, Xu Tan, Zehua Chen, Hongteng Xu, and Guodong Sui. 2024. Tiva: Time-aligned video- to-audio generation. In Proceedings of the 32nd ACM International Conference on Multimedia. 573–582

2024
[42]

Zehan Wang, Yang Zhao, Haifeng Huang, Jiageng Liu, Aoxiong Yin, Li Tang, Linjun Li, Yongqi Wang, Ziang Zhang, and Zhou Zhao. 2023. Connecting multi-modal contrastive representations. Advances in Neural Information Processing Systems 36 (2023), 22099–22114

2023
[43]

Roy Sheffer and Yossi Adi. 2023. I hear your true colors: Image guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 1–5

2023
[44]

Yuhan Shen, Huiyu Wang, Xitong Yang, Matt Feiszli, Ehsan Elhamifar, Lorenzo Torresani, and Effrosyni Mavroudi. 2024. Learning to Segment Referred Objects from Narrated Egocentric Videos. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024), 14510–1...

2024
[45]

Haoran Xu, Jie Zhou, Mengduo Yang, and Jiaze Li. 2024. Shortform ugc video quality assessment based on multi-level video fusion with rank-aware. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , Vol. 7

2024
[46]

arXiv preprint arXiv:2407.05782 (2024)

Sequential contrastive audio-visual learning. arXiv preprint arXiv:2407.05782 (2024)

2024 arXiv
[47]

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

2017
[48]

Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. 2024. Instancediffusion: Instance-level control for im- age generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6232–6242

2024
[51]

Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. 2024. Sonicvi- sionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26866–26875

2024
[52]

Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen
[53]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7151–7161
[55]

Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. 2023. Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.11968 (2023)

2023 arXiv
[56]

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision . 6023–6032

2019
[57]

Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. 2024. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494 (2024)

2024 arXiv
[2019]

In Proceedings of the IEEE/CVF international conference on computer vision

Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision . 6202–6211
[2020]

In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Vggsound: A large-scale audio-visual dataset. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 721–725

2020
[2024]

arXiv preprint arXiv:2402.04825 (2024)

Fast timing-conditioned latent audio diffusion. arXiv preprint arXiv:2402.04825 (2024)

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Gehui Chen, Guan’an Wang, Xiaowen Huang, and Jitao Sang. 2024. Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model. arXiv preprint arXiv:2404.16305 (2024)

arXiv 2024

[2] [2]

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman

work page

[3] [3]

Lihui Chen, Gemine Vivone, Zihao Nie, Jocelyn Chanussot, and Xi- aomin Yang. 2023. Spatial data augmentation: Improving the general- ization of neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing 61 (2023), 1–11

work page 2023

[4] [4]

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. 2023. Tracking anything with decoupled video segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1316–1326

work page 2023

[5] [5]

Jinhao Duan, Quanfu Fan, Hao Cheng, Xiaoshuang Shi, and Kaidi Xu. 2023. Improve video representation with temporal adversarial Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation , , Figure 9. Ablation study of Random Video Stitching (RVS), demonstrating its role in enabling the model to generate distinct sounds corresponding to multipl...

work page Pith review arXiv 2023

[6] [6]

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huam- ing Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 1–5

work page 2023

[7] [7]

Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons

work page

[8] [8]

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He

work page