Pith. sign in

REVIEW 2 major objections 5 minor 3 cited by

Towards Open-Vocabulary Audio-Visual Event Localization

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces open-vocabulary audio-visual event localization, with a 24,800-video benchmark and baselines showing that fine-tuning beats zero-shot on both seen and unseen event classes.

desk verdict New benchmark and task worth engaging, but the headline fine-tuning advantage is confounded by the fusion rule. read the letter →

arxiv 2411.11278 v3 pith:XCNSMZOX submitted 2024-11-18 cs.CV cs.MM

Jinxing Zhou , Dan Guo , Ruohao Guo , Yuxin Mao , Jingjing Hu , Yiran Zhong , Xiaojun Chang , Meng Wang This is my paper · ORCID
classification cs.CVcs.MM
keywords open-vocabularyaudio-visualeventlocalizationOV-AVEBenchdatasettemporalcorrespondencezero-shotmultimodalembeddingfine-tuningbaselinesegment-levelannotationbenchmarkmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a new task, Open-Vocabulary Audio-Visual Event Localization (OV-AVEL): given a 10-second video, a model must mark which one-second segments contain an event that is both audible and visible, and must name the event category even when that category never appeared in training. To support the task, the authors build OV-AVEBench, a dataset of 24,800 videos covering 67 real-life event classes, with 46 classes seen in training and 21 held out as unseen, all manually annotated segment by segment. They also define three evaluation metrics: accuracy, segment-level F1, and event-level F1. The paper argues that a simple fine-tuning baseline, which adds lightweight temporal layers on top of a frozen joint audio-visual-text embedding model, substantially outperforms a training-free zero-shot baseline on both seen and unseen test data, reaching a 57.8 average metric against 46.6. If the result holds, it establishes that open-vocabulary temporal event localization is feasible and gives the community a benchmark to build on.

What carries the argument

The load-bearing object is OV-AVEBench, with its fixed split of 46 seen and 21 unseen classes, its manual segment-level labels, and its three metrics. The method-side mechanism is a joint multimodal embedding: a pretrained model that maps audio, image, and text into one shared space. The training-free baseline computes per-segment cosine similarities between audio features and all candidate class texts, and between visual features and the same texts, then labels a segment as an event only if both modalities pick the same category. The fine-tuning baseline inserts a stack of Transformer layers (one layer works best) after the frozen audio and image encoders to model temporal context, fuses modalities through the geometric mean $\sqrt{S'_{ae}\odot S'_{ve}}$ of audio-text and visual-text probabilities, and adds a special "other" text to absorb background and out-of-list events. The "other" text alone is worth a 10.8-point average gain, and the geometric-mean fusion outperforms simple averaging or feature averaging.

What would settle it

Re-annotate a random sample of test segments with two independent expert annotators and measure per-segment label agreement; if agreement is far below the 46.6-to-57.8 difference between the baselines, the central comparison could be an artifact of noisy ground truth.

Watch

Extended reading notes

Core claim

The central claim is that OV-AVEL is a tractable problem and that OV-AVEBench is a usable testbed for it. The authors report that a training-free baseline, which picks the highest audio-text and visual-text cosine similarity from a joint embedding space and requires the two modalities to agree, reaches 46.6 average on the full test set. Adding one learnable transformer temporal layer per modality on frozen audio and image encoders, then fine-tuning on 13,182 training videos whose classes are all seen, raises the average to 57.8, with gains on both seen (62.9 versus 45.5) and unseen (55.8 versus 47.0) test data. The paper interprets the gains on unseen classes as evidence that temporal-relation learning transfers across categories, and the event-level F1 jump from 34.0 to 49.5 as better temporal boundary localization. The benchmark also shows that closed-set models, which perform well on seen classes, collapse on unseen classes, motivating the open-vocabulary formulation.

Load-bearing premise

The crowd-sourced segment labels in OV-AVEBench are accurate enough to ground every comparison, but the paper reports no agreement or quality statistics for those labels.

Editorial extensions

If this is right

  • If OV-AVEBench is adopted, closed-set AVEL models can be re-evaluated for their unseen-class behavior rather than only their overall accuracy.
  • Open-vocabulary localization can be bootstrapped without retraining the frozen multimodal encoder; adding temporal layers and fine-tuning on seen data is enough to improve unseen-class recognition.
  • Event-level F1 becomes a meaningful third metric, penalizing poor temporal boundaries that segment accuracy alone misses.
  • The benchmark's roughly 3:7 seen-to-unseen test ratio forces models to generalize rather than memorize, and closed-set methods land in the 39-42 average range on total data.
  • Fine-tuning benefits unseen classes mainly by improving temporal boundary localization, since the largest gain appears in event-level F1.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same recipe of frozen joint embeddings plus a small trainable temporal head could transfer to longer videos or to weakly labeled web data, where per-segment manual labels are unavailable.
  • Also beyond the paper, the finding that 25% of training data nearly matches 100% suggests the bottleneck may be the quality of the joint embedding space and the temporal prior, not the size of the seen-class label set; varying the number of seen classes would be a sharper test.
  • Because the paper reports no inter-annotator agreement statistics, an editorial next step is to add a small expert-relabeled subset to OV-AVEBench and measure how much of the 11.2-point fine-tuning gain survives label noise.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces the Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) problem, in which a model must localize audio-visual events in a video and classify them into explicit categories, including categories whose labels do not appear in the training data. To support this task, the authors build OV-AVEBench, a dataset of 24,800 YouTube videos from VGGSound, with manual segment-level labels for 67 event classes (46 seen, 21 unseen during training), and propose three metrics: accuracy, segment-level F1, and event-level F1. They evaluate two baselines: a training-free approach that uses ImageBind embeddings and checks agreement between the top audio-text and visual-text classes per segment, and a fine-tuning approach that adds lightweight temporal transformer layers and fuses audio and visual probabilities with a geometric mean. In their experiments the fine-tuning baseline achieves 57.8 average metric on the total test set versus 46.6 for the training-free baseline, and ablations examine the 'other' text, fusion rule, temporal layers, and training-data ratios.

Significance. If the benchmark and results hold up, OV-AVEBench is a potentially useful resource: it is roughly six times larger than the existing AVE dataset, spans 67 classes, provides segment-level labels, and defines an open-vocabulary evaluation protocol with multiple metrics. The paper also demonstrates a simple zero-shot baseline with ImageBind and that adding a small amount of fine-tuning on seen classes can improve unseen-class localization at the event level. The ablation study on the fusion rule (Table 4) is informative for the community, even though it complicates the headline comparison. The code release supports reproducibility, and the dataset, if reliable, could serve as a standardized testbed for future open-vocabulary audio-visual event localization research.

major comments (2)
  1. [§3.1, §3.2, Table 2, Table 4] The central comparison in Table 2 between the training-free baseline (46.6 Avg.) and the fine-tuning baseline (57.8 Avg.) varies two factors at once: (i) the use of fine-tuned temporal layers and (ii) the rule for combining audio and visual predictions. The training-free model (Sec. 3.1) takes the top-1 class per modality and requires agreement, while the fine-tuning model (Sec. 3.2) uses the geometric-mean fusion of Eq. (2). Table 4 shows that replacing Sqrt with Prob-avg within the fine-tuned model drops Avg. from 57.8 to 39.0, an effect of 18.8 points, which is larger than the 11.2-point gap in Table 2. Since the hard-agreement rule used in the training-free baseline is not directly evaluated with the fine-tuned features, the improvement attributed to 'fine-tuning' could in principle be driven by the fusion rule rather than by learning temporal relations. Please add an ablation that decouples these factors, for example applying the geometric-mean fusion to the training-free model, or applying the hard-consistency rule to the features with temporal layers, and discuss the resulting attribution.
  2. [§2.2] The OV-AVEBench benchmark rests on manual segment-level annotations, but the paper does not report any quantification of label reliability. Section 2.2 describes a two-round review with discussion, yet it gives no inter-annotator agreement (e.g., Cohen's kappa or a per-class agreement rate) and no estimate of label noise. Given that all model evaluations are computed against these labels, the benchmark's usefulness depends on their accuracy. I request that the authors provide agreement statistics on a held-out subset, describe the adjudication process in more detail, and, if possible, report the stability of the main results under a small amount of simulated label noise.
minor comments (5)
  1. [§4.3] In Section 4.3, the in-text reference 'Table A8' should be 'Table 3' for the main-text ablation on the 'other' class, since Table A8 is in the supplementary material.
  2. [§2.1] There are typos: 'downloadeded' in Section 2.1, and the legend in Figure 2(a) appears to repeat 'musical instruments'.
  3. [§4.2] The sentence 'the training-free baseline model performs slightly better on the unseen test data' is ambiguous; if it means that the training-free model performs better on unseen than on seen data, it should be rewritten to avoid the apparent contradiction with Table 2, where the seen Avg. is 45.5 and the unseen Avg. is 47.0.
  4. [§4.2] The paper uses 'significantly outperforms' without reporting variance across runs or significance tests; please clarify whether 'significantly' means statistically tested or merely numerically higher, and report mean and standard deviation if multiple runs were conducted.
  5. [Table 6] The 'Best epoch' column in Table 6 is not explained; please clarify whether the best epoch is selected on the validation set and report the corresponding validation metrics.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark construction, baselines, and evaluation are independent of the claims they support.

full rationale

The paper's load-bearing elements are an externally sourced pretrained model (ImageBind), a manually annotated dataset derived from VGGSound, and standard evaluation metrics (accuracy and F1). The training-free baseline computes ImageBind audio-text and visual-text similarities and applies a hard consistency rule, while the fine-tuning baseline adds learnable temporal layers and uses a geometric-mean fusion; neither baseline's output is defined in terms of the headline claim. Unseen classes are held out of the fine-tuning label set, so the reported unseen performance is not forced by construction. The paper's self-citations (e.g., PSP, contrastive positive sample propagation) appear only as compared prior methods or related work, not as justifications for the central result. The main comparison does confound fine-tuning with a change in fusion rule (Table 4 shows the Sqrt fusion has a large effect), but that is an experimental-design concern, not a circular derivation. No equation reduces to its own input, no fitted parameter is renamed as a prediction, and no load-bearing authority is imported from the authors' prior work.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The central claim rests mainly on the quality of the new dataset and the choice of ImageBind as a frozen backbone, plus the 'other' text prompt for background handling. No continuous free parameters are fitted to derive the claims; the model hyperparameters are standard training choices.

assumptions (3)
  • domain assumption ImageBind's joint embedding space accurately captures semantic alignment between audio, visual, and text modalities for event classification.
    The baselines rely entirely on ImageBind embeddings to compare segment-level audio and visual features with text class names. This is invoked in Sections 1 and 3.1.
  • domain assumption The 67 selected VGGSound categories represent common real-life audio-visual events and are correctly filtered to avoid ambiguity or rarity.
    The dataset construction in Section 2.1 manually filters VGGSound categories by hand, assuming these choices yield a useful benchmark for open-vocabulary localization.
  • ad hoc to paper The 'other' text can adequately represent background and unknown events in the embedding space.
    The special text 'other' is introduced in Section 3.1 to handle background segments and events outside the candidate list. Its effectiveness is ablated, but its semantic adequacy is assumed.
invented entities (1)
  • other text prompt
    purpose: Serve as a textual prototype for the background class and for events not in the candidate list during classification
    The paper adds a special text 'other' to the class list in Section 3.1. It is a design choice, not a measured physical entity; its effectiveness is shown in Table A8 but no external evidence supports its semantic interpretation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Open-Vocabulary Audio-Visual Event Localization." pith.science (2026). https://pith.science/paper/XCNSMZOX

@misc{pith2026241111278,
  author       = {Pith},
  title        = {Pith review of: Towards Open-Vocabulary Audio-Visual Event Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XCNSMZOX}},
  note         = {Machine review of arXiv:2411.11278}
}
read the original abstract

The Audio-Visual Event Localization (AVEL) task aims to temporally locate and classify video events that are both audible and visible. Most research in this field assumes a closed-set setting, which restricts these models' ability to handle test data containing event categories absent (unseen) during training. Recently, a few studies have explored AVEL in an open-set setting, enabling the recognition of unseen events as ``unknown'', but without providing category-specific semantics. In this paper, we advance the field by introducing the Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) problem, which requires localizing audio-visual events and predicting explicit categories for both seen and unseen data at inference. To address this new task, we propose the OV-AVEBench dataset, comprising 24,800 videos across 67 real-life audio-visual scenes (seen:unseen = 46:21), each with manual segment-level annotation. We also establish three evaluation metrics for this task. Moreover, we investigate two baseline approaches, one training-free and one using a further fine-tuning paradigm. Specifically, we utilize the unified multimodal space from the pretrained ImageBind model to extract audio, visual, and textual (event classes) features. The training-free baseline then determines predictions by comparing the consistency of audio-text and visual-text feature similarities. The fine-tuning baseline incorporates lightweight temporal layers to encode temporal relations within the audio and visual modalities, using OV-AVEBench training data for model fine-tuning. We evaluate these baselines on the proposed OV-AVEBench dataset and discuss potential directions for future work in this new field.

Figures

Figures reproduced from arXiv: 2411.11278 by the authors.

Figure 1
Figure 1. (a) Illustration of the AVEL task, which aims to tem￾porally localize segments containing events that are both audible and visible, and identify their categories. (b) Studies of AVEL in different settings. In contrast to previous closed-set and open-set settings, we explore a more practical open-vocabulary AVEL prob￾lem, which needs to infer explicit event categories for both seen and unseen test data (i.e., data co… view at source ↗
Figure 2
Figure 2. Statistics about the proposed OV-AVEBench dataset. (a) Our OV-AVEBench contains 24,800 videos covering 67 practical audio-visual scenes from the real world. Each event category and its corresponding video amount are listed. The category highlighted by a black bounding box indicates that data in that category is only available during the inference phase (unseen classes/data). (b) The audio-visual events in the videos… view at source ↗
Figure 3
Figure 3. Overview of the proposed baseline methods. We utilize the audio and image encoders of the pretrained Imagebind [11] (with frozen parameters) to extract segment-level audio and visual features. ① The training-free baseline sends texts of all candidate classes (both seen and unseen) to extract features. Then, the audio-visual event prediction is decided by evaluating the consistency between audio￾text and visual-text … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Patch-level Sounding Object Tracking for Audio-Visual Question Answering

    cs.MM 2024-12 conditional novelty 7.0 of 10

    A new patch-level sounding object tracking method with motion-, sound-, and question-driven graph modules achieves 78.42% average accuracy on MUSIC-AVQA, competitive with large-scale pretraining approaches.

  2. Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration

    cs.CV 2024-12 conditional novelty 5.0 of 10

    CCNet combines cross-modal consistency and multi-temporal granularity modules to achieve state-of-the-art dense audio-visual event localization on UnAV-100.

  3. Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A class-aware feature decoupling module with a background class plus co-occurrence and local-global fusion blocks improves weakly-supervised audio-visual video parsing.

Reference graph

Works this paper leans on

59 extracted references · 49 canonical work pages · cited by 3 Pith papers

  1. [1]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, pages 609–617, 2017. 2

    work page 2017

  2. [2]

    Objects that sound

    Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, pages 435–451, 2018. 2

    work page 2018

  3. [3]

    Cross-modal label contrastive learning for unsupervised audio-visual event localization

    Peijun Bao, Wenhan Yang, Boon Poh Ng, Meng Hwa Er, and Alex C Kot. Cross-modal label contrastive learning for unsupervised audio-visual event localization. InAAAI, pages 215–222, 2023. 12

    work page 2023

  4. [4]

    VGGSound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zis- serman. VGGSound: A large-scale audio-visual dataset. In ICASSP, pages 721–725, 2020. 3

    work page 2020

  5. [5]

    Localizing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Na- grani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In CVPR, pages 16867–16876,

    work page

  6. [6]

    Cm-pie: Cross-modal per- ception for interactive-enhanced audio-visual video parsing

    Yaru Chen, Ruohao Guo, Xubo Liu, Peipei Wu, Guangyao Li, Zhenbo Li, and Wenwu Wang. Cm-pie: Cross-modal per- ception for interactive-enhanced audio-visual video parsing. In ICASSP, pages 8421–8425, 2024. 1

    work page 2024

  7. [7]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 6, 7, 12

    arXiv 2024

  8. [8]

    Audio-visual event localization via re- cursive fusion by joint co-attention

    Bin Duan, Hao Tang, Wei Wang, Ziliang Zong, Guowei Yang, and Yan Yan. Audio-visual event localization via re- cursive fusion by joint co-attention. In WACV, pages 4013– 4022, 2021. 2

    work page 2021

Show all 59 references
  1. [9]

    Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception

    Junyu Gao, Mengyuan Chen, and Changsheng Xu. Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception. In CVPR, pages 18827–18836, 2023. 1

    2023

  2. [10]

    Learning event-specific localization preferences for audio-visual event localization

    Shiping Ge, Zhiwei Jiang, Yafeng Yin, Cong Wang, Zifeng Cheng, and Qing Gu. Learning event-specific localization preferences for audio-visual event localization. InACM MM, pages 3446–3454, 2023. 12

    2023

  3. [11]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, pages 15180–15190, 2023. 2, 5, 6, 7, 11

    2023

  4. [12]

    Audio-visual instance segmentation

    Ruohao Guo, Xianghua Ying, Yaru Chen, Dantong Niu, Guangyao Li, Liao Qu, Yanyu Qi, Jinxing Zhou, Bowei Xing, Wenzhen Yue, Ji Shi, Qixun Wang, Peiliang Zhang, and Buwen Liang. Audio-visual instance segmentation. arXiv preprint arXiv:2310.18709, 2023. 1

    arXiv

  5. [13]

    Instance-level panoramic audio-visual saliency detection and ranking

    Ruohao Guo, Dantong Niu, Liao Qu, Yanyu Qi, Ji Shi, Wen- zhen Yue, Bowei Xing, Taiyan Chen, and Xianghua Ying. Instance-level panoramic audio-visual saliency detection and ranking. In ACM MM, pages 9426–9434, 2024. 1

    2024

  6. [14]

    Open- vocabulary audio-visual semantic segmentation

    Ruohao Guo, Liao Qu, Dantong Niu, Yanyu Qi, Wenzhen Yue, Ji Shi, Bowei Xing, and Xianghua Ying. Open- vocabulary audio-visual semantic segmentation. In ACM MM, pages 7533–7541, 2024. 1

    2024

  7. [15]

    Unitr: A unified transformer-based framework for co-object and multi-modal saliency detection

    Ruohao Guo, Xianghua Ying, Yanyu Qi, and Liao Qu. Unitr: A unified transformer-based framework for co-object and multi-modal saliency detection. IEEE transactions on multi- media, 2024. 1

    2024

  8. [16]

    Improving audio-visual segmentation with bidirectional generation

    Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong. Improving audio-visual segmentation with bidirectional generation. In AAAI, pages 2067–2075,

  9. [17]

    Cace- net: Co-guidance attention and contrastive enhancement for effective audio-visual event localization

    Xiang He, Xiangxi Liu, Yang Li, Dongcheng Zhao, Guobin Shen, Qingqun Kong, Xin Yang, and Yi Zeng. Cace- net: Co-guidance attention and contrastive enhancement for effective audio-visual event localization. arXiv preprint arXiv:2408.01952, 2024. 2

    2024 arXiv

  10. [18]

    Tri-ergon: Fine-grained video-to-audio generation with multi-modal conditions and lufs control

    Bingliang Li, Fengyu Yang, Yuxin Mao, Qingwen Ye, Hongkai Chen, and Yiran Zhong. Tri-ergon: Fine-grained video-to-audio generation with multi-modal conditions and lufs control. arXiv preprint arXiv:2412.20378, 2024. 1

    2024 arXiv

  11. [19]

    Learning to answer questions in dynamic audio-visual scenarios

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji- Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In CVPR, pages 19108– 19118, 2022. 1

    2022

  12. [20]

    Progressive spatio- temporal perception for audio-visual question answering

    Guangyao Li, Wenxuan Hou, and Di Hu. Progressive spatio- temporal perception for audio-visual question answering. In ACM MM, pages 7808–7816, 2023

    2023

  13. [21]

    Object-aware adaptive-positivity learning for audio- visual question answering

    Zhangbin Li, Dan Guo, Jinxing Zhou, Jing Zhang, and Meng Wang. Object-aware adaptive-positivity learning for audio- visual question answering. In AAAI, pages 3306–3314, 2024

    2024

  14. [22]

    Patch-level sounding object track- ing for audio-visual question answering

    Zhangbin Li, Jinxing Zhou, Jing Zhang, Shengeng Tang, Kun Li, and Dan Guo. Patch-level sounding object track- ing for audio-visual question answering. arXiv preprint arXiv:2412.10749, 2024. 1

    2024 arXiv

  15. [23]

    Dual- modality seq2seq network for audio-visual event localiza- tion

    Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. Dual- modality seq2seq network for audio-visual event localiza- tion. In ICASSP, pages 2002–2006, 2019. 2

    2002

  16. [24]

    Ave-clip: Audioclip-based multi-window temporal transformer for au- dio visual event localization

    Tanvir Mahmud and Diana Marculescu. Ave-clip: Audioclip-based multi-window temporal transformer for au- dio visual event localization. In WACV, pages 5158–5167,

  17. [25]

    T-vsl: Text-guided visual sound source localization in mixtures

    Tanvir Mahmud, Yapeng Tian, and Diana Marculescu. T-vsl: Text-guided visual sound source localization in mixtures. In CVPR, pages 26742–26751, 2024. 1

    2024

  18. [26]

    Contrastive conditional la- tent diffusion for audio-visual segmentation

    Yuxin Mao, Jing Zhang, Mochu Xiang, Yunqiu Lv, Yi- ran Zhong, and Yuchao Dai. Contrastive conditional la- tent diffusion for audio-visual segmentation. arXiv preprint arXiv:2307.16579, 2023. 1

    2023 arXiv

  19. [27]

    Multimodal variational auto-encoder based audio-visual segmentation

    Yuxin Mao, Jing Zhang, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Multimodal variational auto-encoder based audio-visual segmentation. In ICCV, pages 954–965, 2023. 1

    2023

  20. [28]

    Tavg- bench: Benchmarking text to audible-video generation

    Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Tavg- bench: Benchmarking text to audible-video generation. In ACM MM, pages 6607–6616, 2024. 1

    2024

  21. [29]

    Avgzslnet: Audio-visual gen- eralized zero-shot learning by reconstructing label features from multi-modal embeddings

    Pratik Mazumder, Pravendra Singh, Kranti Kumar Parida, and Vinay P Namboodiri. Avgzslnet: Audio-visual gen- eralized zero-shot learning by reconstructing label features from multi-modal embeddings. In WACV, pages 3090–3099,

  22. [30]

    Temporal and cross-modal at- tention for audio-visual zero-shot learning

    Otniel-Bogdan Mercea, Thomas Hummel, A Sophia Koepke, and Zeynep Akata. Temporal and cross-modal at- tention for audio-visual zero-shot learning. In ECCV, pages 488–505. Springer, 2022

    2022

  23. [31]

    Audio-visual generalised zero-shot learning with cross-modal attention and language

    Otniel-Bogdan Mercea, Lukas Riesch, A Koepke, and Zeynep Akata. Audio-visual generalised zero-shot learning with cross-modal attention and language. In CVPR, pages 10553–10563, 2022. 2

    2022

  24. [32]

    Localizing visual sounds the easy way

    Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In ECCV, pages 218–234. Springer, 2022. 1

    2022

  25. [33]

    Audio-visual gener- alized zero-shot learning the easy way

    Shentong Mo and Pedro Morgado. Audio-visual gener- alized zero-shot learning the easy way. arXiv preprint arXiv:2407.13095, 2024. 2

    2024 arXiv

  26. [34]

    Pg-video-llava: Pixel grounding large video- language models

    Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-llava: Pixel grounding large video- language models. arXiv preprint arXiv:2311.13435 , 2023. 12

    2023 arXiv

  27. [35]

    Coordinated joint multimodal embeddings for gen- eralized audio-visual zero-shot classification and retrieval of videos

    Kranti Parida, Neeraj Matiyali, Tanaya Guha, and Gaurav Sharma. Coordinated joint multimodal embeddings for gen- eralized audio-visual zero-shot classification and retrieval of videos. In WACV, pages 3251–3260, 2020. 2

    2020

  28. [36]

    Multiple sound sources localization from coarse to fine

    Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In ECCV, pages 292–308, 2020. 1

    2020

  29. [37]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763, 2021. 5, 7, 11

    2021

  30. [38]

    Fine-grained audible video description

    Xuyang Shen, Dong Li, Jinxing Zhou, Zhen Qin, Bowen He, Xiaodong Han, Aixuan Li, Yuchao Dai, Lingpeng Kong, Meng Wang, et al. Fine-grained audible video description. In CVPR, pages 10585–10596, 2023. 1

    2023

  31. [39]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. In ECCV, pages 247–263, 2018. 1, 2, 5, 7, 12

    2018

  32. [40]

    Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing. In ECCV, pages 436–454, 2020. 1, 2

    2020

  33. [41]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 1–11, 2017. 6

    2017

  34. [42]

    Dual attention matching for audio-visual event localization

    Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. InICCV, pages 6292–6300, 2019. 2

    2019

  35. [43]

    Span-based audio-visual localization

    Yiling Wu, Xinfeng Zhang, Yaowei Wang, and Qingming Huang. Span-based audio-visual localization. In ACM MM, pages 1252–1260, 2022. 2

    2022

  36. [44]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, pages 1–5,

  37. [45]

    Cross-modal background suppres- sion for audio-visual event localization

    Yan Xia and Zhou Zhao. Cross-modal background suppres- sion for audio-visual event localization. In CVPR, pages 19989–19998, 2022. 2, 12

    2022

  38. [46]

    Cross-modal relation-aware networks for audio-visual event localization

    Haoming Xu, Runhao Zeng, Qingyao Wu, Mingkui Tan, and Chuang Gan. Cross-modal relation-aware networks for audio-visual event localization. In ACM MM, pages 3893– 3901, 2020. 2, 7

    2020

  39. [47]

    Avqa: A dataset for audio- visual question answering on videos

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio- visual question answering on videos. In ACM MM, pages 3480–3491, 2022. 1

    2022

  40. [48]

    MM-Pyramid: Multimodal pyramid attentional network for audio-visual event localization and video pars- ing

    Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng, and Yue- jie Zhang. MM-Pyramid: Multimodal pyramid attentional network for audio-visual event localization and video pars- ing. In ACM MM, pages 6241–6249, 2022. 1, 2, 7, 12

    2022

  41. [49]

    Ope- nA VE: Moving towards open set audio-visual event localiza- tion

    Jiale Yu, Baopeng Zhang, Zhu Teng, and Jianping Fan. Ope- nA VE: Moving towards open set audio-visual event localiza- tion. In ACM MM, pages 1–10, 2024. 2

    2024

  42. [50]

    Multimodal class-aware semantic enhance- ment network for audio-visual video parsing

    Pengcheng Zhao, Jinxing Zhou, Dan Guo, Yang Zhao, and Yanxiang Chen. Multimodal class-aware semantic enhance- ment network for audio-visual video parsing. arXiv preprint arXiv:2412.11248, 2024. 1

    2024 arXiv

  43. [51]

    Positive sample propagation along the audio- visual event line

    Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio- visual event line. In CVPR, pages 8436–8444, 2021. 2, 7

    2021

  44. [52]

    Contrastive pos- itive sample propagation along the audio-visual event line

    Jinxing Zhou, Dan Guo, and Meng Wang. Contrastive pos- itive sample propagation along the audio-visual event line. TPAMI, pages 1–18, 2022. 2, 12

    2022

  45. [53]

    Audio–visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio–visual segmentation. In ECCV, pages 386–403, 2022. 1

    2022

  46. [54]

    Im- proving audio-visual video parsing with pseudo visual labels

    Jinxing Zhou, Dan Guo, Yiran Zhong, and Meng Wang. Im- proving audio-visual video parsing with pseudo visual labels. arXiv preprint arXiv:2303.02344, 2023. 1

    2023 arXiv

  47. [55]

    Audio-visual segmentation with semantics

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Ling- peng Kong, Meng Wang, et al. Audio-visual segmentation with semantics. arXiv preprint arXiv:2301.13190, 2023. 1

    2023 arXiv

  48. [56]

    Label-anticipated event disentan- glement for audio-visual video parsing

    Jinxing Zhou, Dan Guo, Yuxin Mao, Yiran Zhong, Xiaojun Chang, and Meng Wang. Label-anticipated event disentan- glement for audio-visual video parsing. In ECCV, pages 1– 22, 2024. 1

    2024

  49. [57]

    Ad- vancing weakly-supervised audio-visual video parsing via segment-wise pseudo labeling

    Jinxing Zhou, Dan Guo, Yiran Zhong, and Meng Wang. Ad- vancing weakly-supervised audio-visual video parsing via segment-wise pseudo labeling. IJCV, pages 1–22, 2024. 1

    2024

  50. [58]

    Dense audio-visual event lo- calization under cross-modal consistency and multi-temporal granularity collaboration

    Ziheng Zhou, Jinxing Zhou, Wei Qian, Shengeng Tang, Xi- aojun Chang, and Dan Guo. Dense audio-visual event lo- calization under cross-modal consistency and multi-temporal granularity collaboration. arXiv preprint arXiv:2412.12628,

    arXiv

  51. [2024]

    Instruction: For the given 10-second video, divide it into 10 one-second segments. For each segment, if its audio and visual streams describe the same event, assign the label “x

    2 Table A7. Ablation study on the number of temporal layers L. Results are reported on the total test data. L Acc. Seg. Eve. Avg. 1 67.1 56.9 49.5 57.8 2 65.4 56.0 49.2 56.9 3 62.8 54.0 47.3 54.7 Table A8. Ablation study on the employment of the text other. ‘TF’ and ‘FT’ repre...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.