Pith. sign in

REVIEW 4 major objections 5 minor 63 references

UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Temporal-aware, uncertainty-weighted pseudo-labels push weakly-supervised audio-visual video parsing past prior state of the art.

desk verdict UWAV is a solid, honestly reported SOTA entry for weakly-supervised AV video parsing, but under-disclosed thresholds and a mechanism claim that doesn't hold for the audio stream keep it from being fully convincing. read the letter →

arxiv 2505.09615 v1 pith:WSTNVQJ7 submitted 2025-05-14 cs.CV cs.SDeess.AS

Yung-Hsuan Lai , Janek Ebbers , Yu-Chiang Frank Wang , François Germain , Michael Jeffrey Jones , Moitreya Chatterjee This is my paper · ORCID
classification cs.CVcs.SDeess.AS
keywords audio-visualvideoparsingweaklysupervisedlearningpseudo-labelgenerationtemporaltransformersuncertaintyweightingfeaturemixupvideo-levellabelseventlocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles weakly supervised audio-visual video parsing (AVVP), where training data supply only video-level event labels and a model must localize events per one-second segment in audio, video, or both. UWAV argues that prior pseudo-label generators are limited because they label each segment in isolation, ignoring inter-segment context, and because they bias toward the majority "event absent" class. To fix this, UWAV pretrains transformer-based pseudo-label generators on a large supervised audio-visual event dataset, using CLIP/CLAP text embeddings so the generators can later be applied to new event vocabularies. On the target dataset, it trains the standard HAN inference module with temporally coherent pseudo-labels weighted by the generator's confidence, a class-rebalancing loss, and feature mixup. The paper reports state-of-the-art F-scores on LLP and top accuracy on AVE, with the largest relative gains on visual-only events.

What carries the argument

The load-bearing object is the temporally coherent pseudo-label generator: two transformer encoders (visual and audio) pretrained on UniAV and then frozen. A segment's center frame goes through CLIP's image encoder; the audio waveform goes through CLAP's audio encoder; each transformer attends across segments via self-attention, and the resulting features are matched to class text embeddings ("A photo of ..." / "This is the sound of ..."). The pretraining loss is BCE on the product of the audio and visual probabilities, which is what makes the two streams share burden while training on audio-visual-only labels. The uncertainty mechanism is a soft target $\hat p_t = \mathrm{Sigmoid}(\hat z_t - \theta) \odot y$: distance from a per-class threshold encodes confidence, and video-level labels mask out classes absent from the video. The mixup regularization interpolates segment features and the same soft targets, giving the model extra training points whose labels are weighted by generator confidence.

What would settle it

On the LLP test set, compare thresholded pseudo-labels from UWAV's frozen generators against per-segment ground truth, separated into audio-only, visual-only, and audio-visual segments; if the generators' accuracy on unimodal segments is no better than VALOR's segment-wise CLIP/CLAP pseudo-labels, then the claimed benefit from temporal coherence and cross-modal pretraining is not actually carrying the result, and downstream gains would have to come from the training-side components instead.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in weakly supervised AVVP is pseudo-label quality, and that quality improves by generating pseudo-labels with a temporal model rather than frame- or segment-wise foundation model scoring. UWAV pretrains a pair of small transformers, one for video and one for audio, on UniAV, where supervision is applied to the elementwise product of audio and visual probabilities because labels are audio-visual; this forces each stream to learn from the other during pretraining. At inference on the target dataset, the frozen transformers encode all segments jointly, so predicted labels for a segment reflect neighboring context; these are combined with video-level ground truth and converted into soft targets by a sigmoid of the margin to a class-wise threshold. Training the HAN inference module with those uncertainty-weighted targets, class-frequency reweighting, and uncertainty-weighted feature mixup yields the reported state-of-the-art numbers, especially for visual events. In the paper's own characterization, pseudo-labels improve by up to 6% Type@AV F-score over prior pseudo-label generators, and the downstream gains follow from that.

Load-bearing premise

The whole pipeline rests on the assumption that a transformer trained on the large UniAV dataset, with labels that only say whether an event is present in both audio and video, learns a text-aligned feature space that remains accurate enough for the smaller LLP and AVE datasets and for event names it never saw during pretraining.

Editorial extensions

If this is right

  • On LLP, the method reports segment-level Type@AV 65.9 and event-level Type@AV 60.9, beating the closest prior PPL by about one point in each, with the visual-event F-score more than three points higher.
  • On AVE, the method reaches 80.6% segment accuracy against 80.4% for VALOR, so the recipe transfers to a smaller dataset with a different event vocabulary.
  • Pseudo-labels generated by the temporal transformers score up to 6% higher Type@AV F-score than VALOR's segment-wise labels, which is the paper's explanation for the downstream gains.
  • The ablation shows uncertainty weighting contributes about 2 points, class reweighting about 1 point, and feature mixup about 0.8 point of segment-level Type@AV F-score.
  • Because the generator is frozen and text-conditioned, the same pretrained module can be pointed at new event sets by swapping text embeddings, which the AVE experiment demonstrates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the transfer claim would be to keep the generator frozen and evaluate it on datasets whose event names are rarer in CLIP/CLAP training; if text-embedding quality limits pseudo-labels, performance should fall even with identically trained temporal transformers.
  • The product-supervision pretraining (Eq. 9) means each stream is supervised only through both-stream agreement; one would expect the separate visual and audio pseudo-labels to be less reliable for events that are strictly unimodal, which is exactly the case LLP's A/V metrics measure.
  • The same uncertainty-weighted pseudo-label recipe could transfer to other weakly supervised temporal localization tasks, such as video-level action detection, where segment-level pseudo-labels are generated by a frozen temporal model and reweighted by confidence.
  • A stronger claim implicit in the results is that temporal coherence matters more than foundation-model raw scoring; a clean comparison would hold the backbone fixed and sweep transformer depth, predicting Type@AV rises with context length.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. UWAV is a two-stage weakly-supervised audio-visual video parsing (AVVP) method. In Stage 1, per-modality transformer pseudo-label generators are pre-trained on the supervised UnAV dataset using a BCE loss on the product of visual and audio probabilities, then transferred to the LLP and AVE datasets by replacing the class text embeddings with CLIP/CLAP encodings of the target classes and thresholding with class-wise constants. In Stage 2, a HAN inference module is trained with binary and soft pseudo-labels, an uncertainty-weighted feature mixup loss, a class-balanced reweighting of the soft-label loss, and the standard video-level BCE loss. The paper reports state-of-the-art results on LLP (e.g., segment-level Type@AV 65.9, event-level Type@AV 60.9) and on AVE (80.6% accuracy), plus ablations, hyperparameter sensitivity, scalability, and compute-time analyses.

Significance. If the results are reproducible, the paper is a meaningful empirical contribution. It combines several sensible ideas for weakly-supervised AVVP -- temporally coherent transformer-based pseudo-labeling, confidence-weighted soft targets, mixup regularization, and class-frequency rebalancing -- and the full system improves on strong recent baselines on two datasets. The supplementary is thorough, including code, runtime, sensitivity to alpha and W, scalability to fewer training videos/classes, and full-metric ablations; these strengthen the manuscript. The main caveats are that the pseudo-label pretraining objective only supervises the joint audio-visual product, the class-wise thresholds are undisclosed, and Eq. (17) appears to reweight at the video level rather than the segment level, so the mechanism behind the reported gains needs clarification.

major comments (4)
  1. [§4.1, Eq. (9)] The pre-training objective L_temp = BCE(p̂^av, y^av) supervises only the element-wise product p̂^v ⊙ p̂^a. This objective is invariant to how probability mass is split across modalities, so the unimodal visual and audio probabilities used to generate pseudo-labels in Eq. (10) are only weakly constrained; the AVVP cases that matter most -- audio-only, visual-only, and temporally misaligned events -- are exactly the ones that this objective never labels directly. The concern is not hypothetical: Table 3 shows the audio pseudo-label F-score is 78.4, below VALOR's 80.5, while the text claims 'more accurate segment-level pseudo-labels' (Section 5.2). Please add direct evidence that the unimodal streams are reliable (e.g., an auxiliary unimodal supervision term, or per-modality pseudo-label diagnostics), or restrict the claim to the overall Type@AV accuracy and explain why the audio inference stream improves despite worse audio pseudo-labels.
  2. [§4.1, Eq. (10)] The class-wise thresholds θv and θa are called 'pre-defined' but their values and selection procedure are never given, in the main text or the supplementary. This is a free parameter of the pseudo-label pipeline and directly controls the binary labels in Eq. (10). Please report all θ values and state whether they were chosen using segment-level validation labels. If they are tuned on ground-truth segments, the procedure is not purely weakly supervised and should be described as such; if not, give the criterion used.
  3. [§4.4, Eq. (17)] The reweighting in L_w-soft multiplies the BCE terms by the video-level label vector y and (1-y), not by segment-level pseudo-labels. For a class present somewhere in a video, Eq. (17) up-weights the loss of every segment regardless of that segment's pseudo-label; for a class absent from the video, every segment is down-weighted. This contradicts the prose, which says the loss from positive/negative events in the pseudo-labels is rebalanced. Please either correct the formula so the mask is per-segment (e.g., using ŷ or p̂ at the segment level) or justify the video-level mask explicitly; as written, the ablation results in Table 4 cannot be attributed to the intended class-balanced mechanism.
  4. [Tables 1 and 2] No error bars or multiple-seed results are reported. Several decisive margins are small -- the AVE gain over VALOR is 0.2% (80.6 vs 80.4), and the LLP gains over PPL are 1.1 and 1.0 for the two Type@AV metrics. Without variance estimates, the state-of-the-art claim is not substantiated. Please report mean and standard deviation over at least three runs for the main tables.
minor comments (5)
  1. [Table 3] The caption says 'Accuracy' but the entries are F-scores; use 'F-score' or report accuracy separately.
  2. [Supplementary Section 13] The text refers to Table A7b for less training data and Table A7a for fewer classes, but the table labels are swapped; please correct.
  3. [§4.2] The term 'uncertainty-weighted' might better be described as 'confidence-weighted', since Eq. (12) uses confidence as the soft target; consider clarifying that lower-confidence targets produce smaller gradients, which is the mechanism.
  4. [Eqs. (14)-(15)] Please define the dimensions of λ and clarify whether the same λ is shared across modalities or sampled independently per modality for the mixup.
  5. [§5.2] The claim of outperforming VALOR/CoLeaf/LEAP 'by up to 3%' should specify the exact baseline and metric, since the margins differ across rows of Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: evaluation is on held-out test sets and pseudo-labels come from a module pre-trained on a separate supervised dataset.

full rationale

The paper's main claims are supported by an external, held-out evaluation protocol rather than by construction. The pseudo-label generator is pre-trained on the large-scale UniAV dataset using supervised audio-visual joint labels (Eq. 9), and is then applied to the target datasets LLP and AVE; the resulting pseudo-labels are evaluated against ground-truth segment labels on the test set (Table 3), and the final AVVP results are computed on the official held-out test splits (Tables 1 and 2). The uncertainty-weighted soft labels in Eq. 12 are margins between logits and thresholds, not fitted targets that mathematically force the final F-scores. Hyperparameters alpha and W are tuned on validation and their sensitivity is reported in the supplementary material. The only overlapping-author baseline, VALOR [20], is an independently published prior method used as a comparison system, not as a load-bearing justification for UWAV's design choices. The weaker audio pseudo-label accuracy relative to VALOR (78.4 vs 80.5, Table 3) is disclosed in the paper and does not make the derivation circular; it is an empirical transfer limitation, not an assumed conclusion. No equation or argument reduces the claimed predictions to the paper's own fitted inputs, so no circular step is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on three principal free components: the unstated class-wise thresholds that convert CLIP/CLAP logits into pseudo-labels, and the two validation-tuned hyperparameters alpha and W. It also relies on a transfer assumption about the pre-trained temporal transformer and on the use of video-level labels to mask pseudo-labels. No new physical or conceptual entities are introduced.

free parameters (3)
  • Class-wise pseudo-label thresholds theta_v and theta_a = not disclosed; one threshold per class for LLP (25) and AVE (29)
    Eq. (10) and Eq. (12) turn logits into binary and soft pseudo-labels via thresholding; no fitting procedure is given, so thresholds are free parameters tuned outside the described algorithm.
  • Mixup Beta concentration alpha = 1.7
    Sensitivity sweep in Fig. A4; selected by validation performance.
  • Class-reweighting scale W = 0.5
    Sensitivity sweep in Fig. A5; selected by validation performance.
assumptions (3)
  • domain assumption Pre-training on UniAV audiovisual event labels, with supervision only on the product p_v ⊙ p_a (Eq. 9), transfers to the target datasets' different class vocabularies and to unimodal events.
    The frozen pseudo-label generator is applied to LLP/AVE after swapping in new CLIP/CLAP text embeddings; if this transfer fails, the temporal transformer contributes nothing over per-segment CLIP/CLAP zero-shot scoring.
  • ad hoc to paper Soft pseudo-label confidence sigmoid(logit - theta) is a valid measure of pseudo-label uncertainty for weighting the training loss.
    Eq. (12) defines uncertainty as margin to a threshold; no calibration or probabilistic justification is provided.
  • domain assumption The ground-truth video-level label y can be used to mask pseudo-labels (Eq. 10), which assumes video-level absence is reliable and that each present class has some positive segment.
    Standard in the weakly-supervised AVVP setup; if video-level labels are noisy or missing, pseudo-labels inherit the error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing." pith.science (2026). https://pith.science/paper/WSTNVQJ7

@misc{pith2026250509615,
  author       = {Pith},
  title        = {Pith review of: UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSTNVQJ7}},
  note         = {Machine review of arXiv:2505.09615}
}
read the original abstract

Audio-Visual Video Parsing (AVVP) entails the challenging task of localizing both uni-modal events (i.e., those occurring exclusively in either the visual or acoustic modality of a video) and multi-modal events (i.e., those occurring in both modalities concurrently). Moreover, the prohibitive cost of annotating training data with the class labels of all these events, along with their start and end times, imposes constraints on the scalability of AVVP techniques unless they can be trained in a weakly-supervised setting, where only modality-agnostic, video-level labels are available in the training data. To this end, recently proposed approaches seek to generate segment-level pseudo-labels to better guide model training. However, the absence of inter-segment dependencies when generating these pseudo-labels and the general bias towards predicting labels that are absent in a segment limit their performance. This work proposes a novel approach towards overcoming these weaknesses called Uncertainty-weighted Weakly-supervised Audio-visual Video Parsing (UWAV). Additionally, our innovative approach factors in the uncertainty associated with these estimated pseudo-labels and incorporates a feature mixup based training regularization for improved training. Empirical results show that UWAV outperforms state-of-the-art methods for the AVVP task on multiple metrics, across two different datasets, attesting to its effectiveness and generalizability.

Figures

Figures reproduced from arXiv: 2505.09615 by the authors.

Figure 1
Figure 1. A weakly-supervised AVVP task example. Events, considered in this task, might be unimodal or multimodal. Even multimodal events, may not be temporally aligned in the audio and visual modalities, e.g. the cello might only be visible in the first few seconds but might produce music, throughout the video. both these senses (modalities) jointly, to enhance machine perception and understanding of real-world events. Vario… view at source ↗
Figure 2
Figure 2. UWAV framework: In stage 1, pseudo-label generation modules are equipped with the ability to capture temporal relationships between segments by pre-training on a large-scale, supervised, audio-visual event localization dataset. In stage 2, temporally coherent, uncertainty-weighted pseudo-labels, derived from the pre-trained pseudo-label generation module, are used to guide the learning of the inference model (HAN) a… view at source ↗
Figure 3
Figure 3. Comparison between predictions by UWAV and competing AVVP methods on the LLP dataset. “GT”: ground truth [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 55 canonical work pages

  1. [1]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  2. [2]

    Visual scene graphs for audio source separation

    Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian. Visual scene graphs for audio source separation. In ICCV, 2021. 2

    work page 2021

  3. [3]

    Learning audio-visual dynamics using scene graphs for au- dio source separation

    Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. Learning audio-visual dynamics using scene graphs for au- dio source separation. In NeurIPS, 2022. 1

    work page 2022

  4. [4]

    Soundspaces: Audio-visual navigation in 3d environments

    Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vi- cenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. InECCV, 2020. 2

    work page 2020

  5. [5]

    Se- mantic audio-visual navigation

    Changan Chen, Ziad Al-Halah, and Kristen Grauman. Se- mantic audio-visual navigation. In CVPR, 2021

    work page 2021

  6. [6]

    Learning to set waypoints for audio-visual navigation

    Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In ICLR, 2021. 2

    work page 2021

  7. [7]

    iquery: Instruments as queries for audio-visual sound separation

    Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang, Ziyao Zeng, and Jianbo Shi. iquery: Instruments as queries for audio-visual sound separation. In CVPR, 2023. 1, 2

    work page 2023

  8. [8]

    Joint-modal label denoising for weakly-supervised audio-visual video parsing

    Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang. Joint-modal label denoising for weakly-supervised audio-visual video parsing. In ECCV,

    work page

Show all 63 references
  1. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 1

    2009

  2. [10]

    Revisit weakly-supervised audio-visual video parsing from the lan- guage perspective

    Yingying Fan, Yu Wu, Yutian Lin, and Bo Du. Revisit weakly-supervised audio-visual video parsing from the lan- guage perspective. In NeurIPS, 2023. 2, 3, 4, 7

    2023

  3. [11]

    Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception

    Junyu Gao, Mengyuan Chen, and Changsheng Xu. Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception. In CVPR, 2023. 7

    2023

  4. [12]

    Audio set: An ontology and human- labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In ICASSP, 2017. 1

    2017

  5. [13]

    Dynamic graph representation learning for video dialog via multi-modal shuffled transformers

    Shijie Geng, Peng Gao, Moitreya Chatterjee, Chiori Hori, Jonathan Le Roux, Yongfeng Zhang, Hongsheng Li, and Anoop Cherian. Dynamic graph representation learning for video dialog via multi-modal shuffled transformers. InAAAI,

  6. [14]

    Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline

    Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. In CVPR, 2023. 2, 4, 5

    2023

  7. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  8. [16]

    Cnn ar- chitectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn ar- chitectures for large-scale audio classification. In ICASSP,

  9. [17]

    Mix and local- ize: Localizing sound sources in mixtures

    Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and local- ize: Localizing sound sources in mixtures. In CVPR, 2022. 2

    2022

  10. [18]

    Egocentric audio-visual object localization

    Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. In CVPR,

  11. [19]

    The kinetics hu- man action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  12. [20]

    Modality-independent teachers meet weakly-supervised audio-visual event parser

    Yung-Hsuan Lai, Yen-Chun Chen, and Frank Wang. Modality-independent teachers meet weakly-supervised audio-visual event parser. In NeurIPS, 2023. 2, 3, 4, 7, 8, 1

    2023

  13. [21]

    Learning to answer questions in dy- namic audio-visual scenarios

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji- Rong Wen, and Di Hu. Learning to answer questions in dy- namic audio-visual scenarios. In CVPR, 2022. 2

    2022

  14. [22]

    Audio-visual segmentation via unlabeled frame exploitation

    Jinxiang Liu, Yikun Liu, Fei Zhang, Chen Ju, Ya Zhang, and Yanfeng Wang. Audio-visual segmentation via unlabeled frame exploitation. In CVPR, 2024. 1, 2

    2024

  15. [23]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 7, 1

    2019

  16. [24]

    Move2hear: Active audio-visual source separation

    Sagnik Majumder, Ziad Al-Halah, and Kristen Grauman. Move2hear: Active audio-visual source separation. In ICCV,

  17. [25]

    Multimodal variational auto-encoder based audio-visual segmentation

    Yuxin Mao, Jing Zhang, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Multimodal variational auto-encoder based audio-visual segmentation. In ICCV, 2023. 2

    2023

  18. [26]

    Multi-modal grouping net- work for weakly-supervised audio-visual video parsing

    Shentong Mo and Yapeng Tian. Multi-modal grouping net- work for weakly-supervised audio-visual video parsing. In NeurIPS, 2022. 2, 3

    2022

  19. [27]

    Audio-visual grouping net- work for sound localization from mixtures

    Shentong Mo and Yapeng Tian. Audio-visual grouping net- work for sound localization from mixtures. In CVPR, 2023. 1, 2

    2023

  20. [28]

    Beyond mono to binaural: Generating binaural au- dio from mono audio with depth and cross modal attention

    Kranti Kumar Parida, Siddharth Srivastava, and Gaurav Sharma. Beyond mono to binaural: Generating binaural au- dio from mono audio with depth and cross modal attention. In WACV, 2022. 2

    2022

  21. [29]

    Weakly-supervised audio-visual video parsing with prototype-based pseudo-labeling

    Kranthi Kumar Rachavarapu, Kalyan Ramakrishnan, et al. Weakly-supervised audio-visual video parsing with prototype-based pseudo-labeling. In CVPR, 2024. 2, 3, 5, 7, 8

    2024

  22. [30]

    Boosting positive seg- ments for weakly-supervised audio-visual video parsing

    Kranthi Kumar Rachavarapu et al. Boosting positive seg- ments for weakly-supervised audio-visual video parsing. In ICCV, 2023. 7

    2023

  23. [31]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 2, 3, 4, 1

    2021

  24. [32]

    Dual perspective network for audio-visual event localization

    Varshanth Rao, Md Ibrahim Khalil, Haoda Li, Peng Dai, and Juwei Lu. Dual perspective network for audio-visual event localization. In ECCV, 2022. 1, 2

    2022

  25. [33]

    Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In CVPR, 2023. 2

    2023

  26. [34]

    Coleaf: A contrastive-collaborative learning framework for weakly supervised audio-visual video pars- ing

    Faegheh Sardari, Armin Mustafa, Philip JB Jackson, and Adrian Hilton. Coleaf: A contrastive-collaborative learning framework for weakly supervised audio-visual video pars- ing. In ECCV, 2024. 7, 1

    2024

  27. [35]

    Sound source local- ization is all about cross-modal alignment

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Sound source local- ization is all about cross-modal alignment. In ICCV, 2023. 1, 2

    2023

  28. [36]

    Audio-visual scene-aware dialog and reasoning using audio- visual transformers with joint student-teacher learning

    Ankit Shah, Shijie Geng, Peng Gao, Anoop Cherian, Takaaki Hori, Tim K Marks, Jonathan Le Roux, and Chiori Hori. Audio-visual scene-aware dialog and reasoning using audio- visual transformers with joint student-teacher learning. In ICASSP, 2022. 2

    2022

  29. [37]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. In ECCV, 2018. 1, 2, 6, 7, 3

    2018

  30. [38]

    Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing. In ECCV, 2020. 1, 2, 3, 6, 7, 8

    2020

  31. [39]

    Cyclic co-learning of sounding object visual grounding and sound separation

    Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In CVPR, 2021. 2

    2021

  32. [40]

    A closer look at spatiotemporal convolutions for action recognition

    Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018. 7, 1

    2018

  33. [41]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 2, 3, 4

    2017

  34. [42]

    Exploring heterogeneous clues for weakly-supervised audio-visual video parsing

    Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In CVPR,

  35. [43]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023. 2, 3, 1

    2023

  36. [44]

    Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners. In CVPR,

  37. [45]

    Cooperation does matter: Exploring multi-order bilateral relations for audio- visual segmentation

    Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. Cooperation does matter: Exploring multi-order bilateral relations for audio- visual segmentation. In CVPR, 2024. 2

    2024

  38. [46]

    Lavss: Location-guided audio-visual spatial audio separation

    Yuxin Ye, Wenming Yang, and Yapeng Tian. Lavss: Location-guided audio-visual spatial audio separation. In WACV, 2024. 1, 2

    2024

  39. [47]

    Catch me if you hear me: Audio-visual navigation in complex unmapped environments with moving sounds

    Abdelrahman Younes, Daniel Honerkamp, Tim Welschehold, and Abhinav Valada. Catch me if you hear me: Audio-visual navigation in complex unmapped environments with moving sounds. IEEE Robotics and Automation Letters, 2023. 2

    2023

  40. [48]

    Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video pars- ing

    Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng, and Yue- jie Zhang. Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video pars- ing. In ACM MM, 2022. 2, 3

    2022

  41. [49]

    Sound adversarial audio- visual navigation

    Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, and Xiaohong Liu. Sound adversarial audio- visual navigation. In ICLR, 2022. 2

    2022

  42. [50]

    Pano-avqa: Grounded audio-visual question answering on 360deg videos

    Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In ICCV, 2021. 2

    2021

  43. [51]

    Positive sample propagation along the audio- visual event line

    Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio- visual event line. In CVPR, 2021. 2

    2021

  44. [52]

    Audio–visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio–visual segmentation. In ECCV, 2022. 1, 2

    2022

  45. [53]

    Im- proving audio-visual video parsing with pseudo visual labels

    Jinxing Zhou, Dan Guo, Yiran Zhong, and Meng Wang. Im- proving audio-visual video parsing with pseudo visual labels. arXiv preprint arXiv:2303.02344, 2023. 2, 3, 4, 7

    2023 arXiv

  46. [54]

    Label-anticipated event dis- entanglement for audio-visual video parsing

    Jinxing Zhou, Dan Guo, Yuxin Mao, Yiran Zhong, Xiao- jun Chang, and Meng Wang. Label-anticipated event dis- entanglement for audio-visual video parsing. arXiv preprint arXiv:2407.08126, 2024. 2, 3, 7 UW A V: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing Sup...

    2024 arXiv

  47. [55]

    Limitations Although UW A V achieves state-of-the-art results on the A VVP task, compared to competing methods, it requires additional training data to pre-train the pseudo-label gen- eration module (on which we train for about 80 epochs)

  48. [56]

    Infer- ence Time

    Implementation Details of UW A V To assess the effectiveness of our method, in line with prior work [38], each 10-second video in both the LLP [38] and A VE [37] datasets is split into 10 segments of one sec- ond each, where each segment contains 8 frames. The visual feature b...

  49. [57]

    Details of Compute Environment Our model is trained on a desktop computer with an Intel Core i7 CPU, with 32GB RAM, and a singleNVIDIA RTX 3090 GPU

  50. [58]

    CoLeaf [34] on the LLP dataset [38]

    Analysis of Compute Time For a more holistic understanding of the performance of our method, we compare its training and inference times with the most recently published approach for the A VVP task, viz. CoLeaf [34] on the LLP dataset [38]. The results of this study are shown ...

  51. [59]

    CLIP and CLAP as visual and audio feature backbones

    Quantitative Comparison Using Better Backbone Features We also quantitatively compare our proposed approach with V ALOR on the LLP dataset using better backbone features, i.e. CLIP and CLAP as visual and audio feature backbones. As shown in Table A6, UW A V outperforms V ALOR ...

  52. [60]

    When α is adjusted, class-balanced loss re-weighting is not applied

    Sensitivity to the Choice of α andW To gain a better understanding of the effect of the choice of hyper-parameters on our model’s performance, we eval- uate the sensitivity of our model to the choice α in the uncertainty-weighted feature mixup and W in the class- balanced loss...

  53. [61]

    The scalability of UW A V

    The Scalability of UW A V To evaluate the scalability of UW A V , we train the inference model (HAN) with less training data (Table A7b) as well as Table A7. The scalability of UW A V . (a) Training with different amounts of data. Training Data Ratio Segment-level A V A V Type...

  54. [62]

    Binary” denotes training with binary pseudo-labels. “Soft

    Ablation Studies Ablation Study on All Metrics: In Table A8, we report the ablation study on all metrics for a more complete un- Table A8. Ablation study reported on all metrics. “Binary” denotes training with binary pseudo-labels. “Soft” denotes training with uncertainty-weig...

  55. [63]

    Figure A8 shows the same, for sample videos on the A VE dataset [37]

    Qualitative Results Figures A6, A7 show event predictions of our method ver- sus competing baselines on sample videos from the LLP dataset [38]. Figure A8 shows the same, for sample videos on the A VE dataset [37]. As is evident from the figures, we see consistently accurate e...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.