Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing

Deng Yu; Guohua Lv; Sijiu Niu; Xiangcheng Sun; Yongbiao Gao

REVIEW 3 major objections 5 minor 46 references

Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Label denoising for audio-visual video parsing should be trained jointly with the parser, using validation F-scores as the reward, and this jointly trained denoiser outperforms prior denoisers and improves existing parsers.

desk verdict Novel RL-based label denoising for AVVP, but the validation-set reward confounds the experimental comparison; needs a revised evaluation to be convincing. read the letter →

arxiv 2412.19563 v1 pith:KIR46Y2Y submitted 2024-12-27 cs.CV

Yongbiao Gao , Xiangcheng Sun , Guohua Lv , Deng Yu , Sijiu Niu This is my paper · ORCID

classification cs.CV

keywords audio-visualvideoparsingweakly-supervisedlearninglabeldenoisingreinforcementREINFORCEmodality-specificnoiseLLPdataseteventlocalization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that label denoising for weakly-supervised audio-visual video parsing should not be a separate preprocessing step. It proposes RLLD, a reinforcement-learning agent that decides which event labels to remove from each modality, trained jointly with the parser so that denoising is guided by the parser's own validation performance. The agent's state combines input features with the original noisy labels, and its reward combines a validation F-score with a soft inter-reward comparing revised labels to smoothed originals. If the central claim holds, denoisers that optimize parsing outcomes directly outperform denoisers that estimate noise in a disconnected stage, and the same denoiser can be dropped into existing parsers to improve them.

What carries the argument

The load-bearing object is the reinforcement-learning label-denoising policy. At each step the state is the concatenation of pretrained audio and visual features with the original video-level labels; self-attention and cross-attention produce hidden states, and a Bernoulli sample over class probabilities decides which labels to delete from each modality. The revised labels supervise both video-level and modality-level losses of a HAN-style task network. Since the validation F-score reward is non-differentiable, REINFORCE with a Monte Carlo estimate updates the policy; the soft inter-reward, built from a smoothed-label divergence and cosine similarity between revised and smoothed labels, supplies immediate feedback, while a terminal reward equal to the parser's validation F-score supplies the task-aligned signal.

What would settle it

Train RLLD with the same validation-reward rule but on a split of the validation set held out from the original validation data, or with permuted validation labels, and compare test F-scores; if deleting labels that raise the validation reward does not also raise test scores, the reported gains come from validation-set memorization rather than better denoising.

Watch

Extended reading notes

Core claim

On the LLP benchmark, RLLD reports segment-level audio F-score 61.6 and event-level Event@Audio-Visual 53.4, above JoMoLD's 60.6 and 51.4; adding a contrastive loss gives 63.4 and 54.0. Integrated into CPSP and VALOR, it raises their reported F-scores on most metrics, for example VALOR segment-level Type from 62.0 to 62.7 and event-level Event from 54.2 to 54.9. The authors attribute the gains to using validation results as a reward, which aligns the denoising policy with the actual parsing objective, and to the soft inter-reward, which provides step-by-step feedback during learning.

Load-bearing premise

The denoiser is trained with a reward equal to the parser's F-score on the validation set, so the method assumes that validation F-score is a trustworthy signal that transfers to unseen test videos rather than being overfit by the policy.

Editorial extensions

If this is right

If RLLD works as claimed, label denoising in audio-visual video parsing can be optimized end-to-end toward parsing performance instead of being a separate noise-estimation stage.
Because the reward is just the parser's validation F-score, the denoiser can in principle be attached to any AVVP parser that has a validation set, not only the HAN backbone used in the paper.
Removing labels rather than estimating noise ratios gives downstream components such as contrastive learning cleaner modality-specific supervision, which may explain the gains when RLLD is added to CPSP and VALOR.
The paper's training-time comparison (about 8 hours versus 2.5-3 hours for MA and JoMoLD) means the F-score improvements come with a substantial compute cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A clean test of the method would hold out a third split of the validation set, since using validation F-scores as a training reward makes the validation set part of the training signal and could inflate test results through selection.
The same reward scheme could transfer to other weakly-supervised sequence labeling problems with noisy per-modality labels, such as action localization in video, because the machinery does not depend on audio-visual specifics.
The ablation results suggest the soft inter-reward acts mainly as a regularizer that keeps the policy close to the original labels; without it performance drops, implying the agent's freedom to delete labels is useful only when tempered.
The paper itself notes (Section 4.6) that reinforcement learning's trial-and-error nature makes training slow, so practical adoption depends on whether the reported F-score gains justify roughly triple the training time of prior denoisers.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The real contribution here is the joint RL denoising loop: a binary action policy removes modality-specific label noise, and the reward comes partly from the parser's F-score on the validation split. That integration is new for AVVP, and the authors are upfront about it. The experiments also show the denoiser can be bolted onto existing parsers like VALOR and CPSP, with small but consistent gains.

The soft spot is the one you'd expect. The denoiser is trained with a terminal reward that is exactly the task network's F-score on the 649-video validation set. That means validation labels leak into training for the denoiser. The baselines (JoMoLD, MA, and the rest) never see those labels during training. So the reported test improvements — a few F-score points — could be partly explained by the extra supervision from validation, not by the RL/joint-training design. The paper doesn't report error bars or significance tests, and the margins are small. The ablation 'w/o soft inter-reward' still uses the validation reward, so it doesn't isolate this effect. What's missing is a version of RLLD that trains without validation rewards, or a comparison against baselines that also get validation labels, or a cross-validation scheme.

I don't think the idea is wrong. The validation reward is a reasonable choice if the goal is a direct signal for denoising, and the paper is transparent about it. But the evidence as presented doesn't yet show that RLLD is better denoising rather than better at exploiting validation labels. That's fixable: run the method on a split where validation labels are never used, or at least report test results with standard deviations over random seeds and a baseline with the same validation supervision.

The paper is worth a serious referee. The framing is clear, the related work is covered, and the method can be repaired with a cleaner evaluation. For a reading group, the validation-leakage discussion is the most instructive piece — it's a nice case study in how an intuitive training signal can compromise a comparison.

I'd recommend sending it to review, with a request for a revised evaluation that isolates the denoiser's contribution.

Referee Report

3 major / 5 minor

Summary. The paper proposes RLLD, a reinforcement-learning-based label denoising module for weakly-supervised audio-visual video parsing (AVVP). A denoising policy is trained with REINFORCE to remove modality-specific noisy labels before these labels are used to train a parsing network; the reward combines an immediate soft inter-reward, measuring consistency between smoothed original labels and revised labels, with a terminal reward equal to the parsing model's F-score on the official validation split. Experiments on the LLP dataset compare RLLD with existing denoisers (JoMoLD, MA) and report small improvements, and the paper also reports additional gains when RLLD is integrated into CPSP and VALOR.

Significance. If the validation-reward protocol can be made sound, the paper would be a useful contribution: it is, to my knowledge, the first RL formulation of label denoising for AVVP, and the joint training of denoiser and parser is a natural way to address the disconnect between denoising and parsing. The integration experiments with CPSP and VALOR are a useful feature, and the paper is transparent about the validation-based reward. However, the central quantitative claim is currently weakened by the training-on-validation protocol and by the absence of uncertainty estimates; these issues are fixable, but they are load-bearing for the claim of superior denoising.

major comments (3)

[Algorithm 1, Eq. (10), Sec. 4.1] The terminal reward in Eq. (10) is the F-score of the task network on the official 649-video validation split, and Algorithm 1 (steps 5-7) uses this reward in the REINFORCE update of Eq. (15). Validation labels therefore act as training labels for the denoising policy, not merely as a model-selection signal. The baselines in Table 1 (e.g., JoMoLD, MA) never receive validation labels during training, so the comparison is not controlled: the reported test improvements (e.g., Event@AV 53.4 vs 51.4 for JoMoLD) could reflect overfitting to the validation split rather than a genuinely better denoising strategy. Please address this by, at minimum, ablating the reward source (e.g., using a held-out portion of the training set for the reward), reporting the validation-versus-test gap for the learned policy, and either giving baselines the same validation-based feedback or explicitly arguing why that feedback would not change their results.
[Table 1, Secs. 4.2 and 4.5] Table 1 reports single F-score values with no standard deviations, number of seeds, or significance tests. The margins over JoMoLD are about 1.0 point at segment-level audio (61.6 vs 60.6), 2.4 points at segment-level visual (64.6 vs 62.2), and 2.0 points at event-level Event@AV (53.4 vs 51.4); these are small relative to typical run-to-run variation in this setting. Since Section 4.5 already reports five random initializations for the convergence curves, the authors have the infrastructure to report mean +/- std for the final test metrics; they should do so and add a suitable significance test before claiming 'superior performance'.
[Table 2, Sec. 4.4] The ablation in Table 2 does not isolate the claimed components: the 'Full Setting' row is 'RLLD+CL', while the ablated rows are 'RLLD w/o initialized labels' and 'RLLD w/o soft inter-reward'. As presented, the ablated variants do not include the CL loss, so the difference attributed to the soft inter-reward (e.g., segment-level audio 61.8 vs 63.4) conflates the reward term with the contrastive loss. Please rerun the ablations on the same base configuration (all variants with or all variants without CL) and report both configurations.

minor comments (5)

[Eq. (9)] The formula for R1 is difficult to parse: the variables l_soft^t and l_t^in(j) are not defined precisely, the indices j and m are unexplained, and the expression as typeset appears to mix sums and division incorrectly. Please rewrite Eq. (9) with clear definitions of every symbol.
[Eq. (13)] The text says 'R(S) is computed by Eq.(9)', but Eq. (9) defines R_inter only; the full reward R(S) is defined in Eq. (11). Please correct the reference.
[Eqs. (6)-(8), Algorithm 1] The video-level loss L_video in Eq. (7) is defined but does not appear in Algorithm 1 or in the training description. Please clarify whether L_video is used and, if so, where.
[Eq. (1)] The state definition mixes indices: s_i_audio uses f^t_audio and y^i_a, while i and t are both used for the video index. Please use one consistent subscript/superscript convention.
[Sec. 4.2] The text says 'Table 3 presents the F-score results', but the main comparison table is Table 1. Please correct the cross-reference.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; validation-reward training is a leakage/overfitting concern, not an in-paper circularity.

full rationale

The central claim is that the RL label-denoiser improves AVVP F-scores on the LLP test set. The denoiser's objective J(θ) (Eq. 12) maximizes R(S), where R_terminal = A(s_a, s_v, a_a, a_v) is the validation F-score (Eq. 10) of the task network. This is a legitimate, if unconventional, training signal: the validation labels enter the reward, while the test-set metrics reported in Table 1 are not used in any update. The denoised labels feed the task network via Eqs. (5)-(8), and the task network in turn provides the reward, forming a coupled training loop rather than a definitional circle. The only self-citation is [12] (SeqLE) for the top-k selection in the VALOR integration; it is a peer-reviewed, external component and is not the source of the central denoising result. The validation-reward design does raise a correctness risk—baselines do not receive validation labels during training, and the margins in Table 1 are small without error bars—but that is an overfitting/leakage concern, not circularity under this analysis. On the paper's own equations, no reported 'prediction' reduces to its input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the modality-specific noise assumption and on the validity of validation-based rewards. Two reward weights and a reward scaling factor are hand-set, and the top-k threshold in the VALOR adaptation is taken from the authors' own prior work.

free parameters (4)

alpha_1 = 0.6
Weight for the KL-divergence soft inter-reward R1 in Eq. (9), chosen by hand.
alpha_2 = 0.4
Weight for the cosine-similarity soft inter-reward R2 in Eq. (9), chosen by hand.
reward_scaling = 0.1
Validation F-scores are scaled by 0.1 before being used as rewards, as stated in Section 4.1.
topk_threshold = 0.5
In the VALOR integration, the sum of top-k pseudo-label probabilities is capped at 0.5, following SeqLE [12], used to select relevant labels.

assumptions (4)

domain assumption Weakly-supervised AVVP provides only video-level labels for training, but segment-level labels for validation.
Stated in Section 3.1; the whole training pipeline depends on this asymmetry.
domain assumption Modality-specific noisy labels (an event present in one modality but absent in the other) degrade parsing and are worth removing.
Adopted from [38, 6] and restated in Section 1; the denoising objective is defined against it.
ad hoc to paper Validation F-score is a suitable reward signal for training the denoising policy.
Section 3.3 and Eq. (10); the method's learning signal is this validation feedback, which risks overfitting.
domain assumption The HAN architecture is a reasonable task network and reward evaluator.
Section 3.3; HAN is used as the task network, and any conclusions are tied to this backbone.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing." pith.science (2026). https://pith.science/paper/KIR46Y2Y

@misc{pith2026241219563,
  author       = {Pith},
  title        = {Pith review of: Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KIR46Y2Y}},
  note         = {Machine review of arXiv:2412.19563}
}

read the original abstract

Audio-visual video parsing (AVVP) aims to recognize audio and visual event labels with precise temporal boundaries, which is quite challenging since audio or visual modality might include only one event label with only the overall video labels available. Existing label denoising models often treat the denoising process as a separate preprocessing step, leading to a disconnect between label denoising and AVVP tasks. To bridge this gap, we present a novel joint reinforcement learning-based label denoising approach (RLLD). This approach enables simultaneous training of both label denoising and video parsing models through a joint optimization strategy. We introduce a novel AVVP-validation and soft inter-reward feedback mechanism that directly guides the learning of label denoising policy. Extensive experiments on AVVP tasks demonstrate the superior performance of our proposed method compared to label denoising techniques. Furthermore, by incorporating our label denoising method into other AVVP models, we find that it can further enhance parsing results.

Figures

Figures reproduced from arXiv: 2412.19563 by the authors.

**Figure 1.** The overall framework of our proposed joint training RLLD for AVVP. The label denoising module aims to generate the denoising [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 4.** Event-level results of VALOR and RLLD+VALOR. Audio Visual Audio-Visual Type Event Event-level 0 10 20 30 40 50 60 70 F-score(%) 51.6 54.0 46.5 50.7 49.9 53.1 55.6 47.9 52.2 51.3 CPSP RLLD+CPSP [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 6.** The audio curve under segment-level during the training. 0 200 400 600 800 1000 episode 5.58 5.60 5.62 5.64 5.66 5.68 5.70 5.72 mean_return_of_the_past_100_eposides Seg-level-visual [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

Figures from the paper (2 more)

**Figure 8.** Figure 8: The event@AV curve under segment-level during the training. 0 200 400 600 800 1000 episode 8.33 8.34 8.35 8.36 8.37 mean_return_of_the_past_100_eposides Event-level-Event@AV [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 10.** Figure 10: The audio curve under event-level during the training. 0 200 400 600 800 1000 episode 3.84 3.86 3.88 3.90 3.92 3.94 3.96 3.98 mean_return_of_the_past_100_eposides Event-level-visual [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 44 canonical work pages

[1]

Arandjelovic and A

R. Arandjelovic and A. Zisserman. Look, listen and learn. In IEEE international conference on computer vision , pages 609–617, 2017. 3

work page 2017
[2]

Aytar, C

Y . Aytar, C. V ondrick, and A. Torralba. Soundnet: Learning sound representations from unlabeled video. Neural Infor- mation Processing Systems, pages 892–900, 2016. 3

work page 2016
[3]

Bayoudh, R

K. Bayoudh, R. Knani, F. Hamdaoui, and A. Mtibaa. A sur- vey on deep multimodal learning for computer vision: ad- vances, trends, applications, and datasets. The Visual Com- puter, pages 1–32, 2021. 1

work page 2021
[4]

D. A. Bulkin and J. M. Groh. Seeing sounds: visual and auditory interactions in the brain. Current Opinion in Neu- robiology, 16(4):415–419, 2006. 1

work page 2006
[5]

H. Chen, D. Zhu, G. Zhang, W. Shi, X. Zhang, and J. Li. Cm- cs: Cross-modal common-specific feature learning for audio- visual video parsing. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 1–5, 2023. 1

work page 2023
[6]

Cheng, Z

H. Cheng, Z. Liu, H. Zhou, C. Qian, W. Wu, and L. Wang. Joint-modal label denoising for weakly-supervised audio- visual video parsing. In European Conference on Computer Vision, pages 431–448. Springer, 2022. 1, 2, 3, 6

work page 2022
[7]

Cheng, R

Y . Cheng, R. Wang, Z. Pan, R. Feng, and Y . Zhang. Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning. In ACM International Conference on Multimedia, pages 3884–3892, 2020. 3

work page 2020
[8]

Ephrat, I

A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein. Look- ing to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. arXiv preprint arXiv:1804.03619, 2018. 3

arXiv 2018

Show all 46 references

[9]

Y . Fan, Y . Wu, Y . Lin, and B. Du. Revisit weakly-supervised audio-visual video parsing from the language perspective. In Neural Information Processing Systems, 2023. 3 9

2023
[10]

J. Gao, M. Chen, and C. Xu. Collecting cross-modal presence-absence evidence for weakly-supervised audio- visual event perception. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 18827–18836,
[11]

Gao and K

R. Gao and K. Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency. InIEEE/CVF Con- ference on Computer Vision and Pattern Recognition , pages 15490–15500, 2021. 3

2021
[12]

Y . Gao, K. Wang, and X. Geng. Sequential label enhance- ment. IEEE Transactions on Neural Networks and Learning Systems, pages 1–12, 2022. 7

2022
[13]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn- ing for image recognition. In IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6

2016
[14]

Hershey, S

S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al. Cnn architectures for large-scale audio classification. In IEEE international conference on acous- tics, speech and signal processing , pages 13...

2017
[15]

A. Hu, F. Cotter, N. Mohan, C. Gurau, and A. Kendall. Prob- abilistic future prediction for video scene understanding. In European conference on computer vision , pages 767–785,
[16]

D. Hu, F. Nie, and X. Li. Deep multimodal clustering for un- supervised audiovisual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9248– 9257, 2019. 3

2019
[17]

R. A. Jacobs and C. Xu. Can multisensory training aid visual learning? a computational investigation. Journal of Vision, 19(11):1–1, 2019. 1

2019
[18]

Jiang, X

X. Jiang, X. Xu, Z. Chen, J. Zhang, J. Song, F. Shen, H. Lu, and H. T. Shen. Dhhn: Dual hierarchical hybrid network for weakly-supervised audio-visual video parsing. In ACM International Conference on Multimedia , pages 719–727,
[19]

S. Lee, J. Chung, Y . Yu, G. Kim, T. Breuel, G. Chechik, and Y . Song. Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning. In IEEE/CVF International Conference on Computer Vision , pages 10274–10284, 2021. 3

2021
[20]

K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y . Lu, and L. Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17949– 17958, 2022. 3

2022
[21]

Lin, Y .-J

Y .-B. Lin, Y .-J. Li, and Y .-C. F. Wang. Dual-modality seq2seq network for audio-visual event localization. InIEEE International Conference on Acoustics, Speech and Signal Processing, pages 2002–2006, 2019. 6

2002
[22]

Lin, H.-Y

Y .-B. Lin, H.-Y . Tseng, H.-Y . Lee, Y .-Y . Lin, and M.-H. Yang. Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing. Neural Infor- mation Processing Systems , pages 11449–11461, 2021. 3, 6

2021
[23]

Ling and X

M. Ling and X. Geng. Soft video parsing by label distribu- tion learning. Frontiers of Computer Science , 13:302–317,
[24]

J. Miao, Y . Wei, Y . Wu, C. Liang, G. Li, and Y . Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In IEEE/CVF conference on computer vision and pattern recognition, pages 4133–4143, 2021. 1

2021
[25]

Mo and Y

S. Mo and Y . Tian. Multi-modal grouping network for weakly-supervised audio-visual video parsing. Neural Infor- mation Processing Systems, pages 34722–34733, 2022. 1, 3, 6

2022
[26]

Nguyen, S

A. Nguyen, S. J ¨arvel¨a, C. Ros ´e, H. J ¨arvenoja, and J. Malm- berg. Examining socially shared regulation and shared phys- iological arousal events with multimodal learning analytics. British Journal of Educational Technology , 54(1):293–312,
[27]

K. K. Rachavarapu et al. Boosting positive segments for weakly-supervised audio-visual video parsing. In IEEE/CVF International Conference on Computer Vision, pages 10192– 10202, 2023. 1, 2, 3, 6

2023
[28]

Rahman, B

T. Rahman, B. Xu, and L. Sigal. Watch, listen and tell: Multi-modal weakly supervised dense event captioning. In IEEE/CVF international conference on computer vision , pages 8908–8917, 2019. 3

2019
[29]

Ramachandram and G

D. Ramachandram and G. W. Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE Signal Processing Magazine, 34(6):96–108, 2017. 1

2017
[30]

Sutton and A

R. Sutton and A. Barto. Reinforcement learning: An intro- duction. IEEE Transactions on Neural Networks, 9(5):1054– 1054, 1998. 6

1998
[31]

Y . Tian, D. Li, and C. Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In European Conference on Computer Vision , pages 436–454. Springer,
[32]

Y . Tian, J. Shi, B. Li, Z. Duan, and C. Xu. Audio-visual event localization in unconstrained videos. InEuropean con- ference on computer vision, pages 247–263, 2018. 6

2018
[33]

F. Tosi, F. Aleotti, P. Z. Ramirez, M. Poggi, S. Salti, L. D. Stefano, and S. Mattoccia. Distilled semantics for compre- hensive scene understanding from videos. InIEEE/CVF con- ference on computer vision and pattern recognition , pages 4654–4665, 2020. 1

2020
[34]

D. Tran, H. Wang, L. Torresani, J. Ray, Y . LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018. 6

2018
[35]

Wang and D

Y . Wang and D. Wang. Towards scaling up classification- based speech separation. IEEE Transactions on Audio, Speech, and Language Processing , 21(7):1381–1390, 2013. 3

2013
[36]

H. Wei, L. Feng, X. Chen, and B. An. Combating noisy labels by agreement: A joint training method with co- regularization. In IEEE/CVF conference on computer vision and pattern recognition, pages 13726–13735, 2020. 6

2020
[37]

Williams and J. Ronald. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Ma- chine Learning, 3-4(8):229–256, 1992. 5

1992
[38]

Wu and Y

Y . Wu and Y . Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1326–1335, 2021. 1, 2, 3, 6 10

2021
[39]

P. Xu, X. Zhu, and D. A. Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 2023. 1

2023
[40]

A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video cap- tioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10714–10726, 2023. 3

2023
[41]

J. Yu, Y . Cheng, R.-W. Zhao, R. Feng, and Y . Zhang. Mm- pyramid: Multimodal pyramid attentional network for audio- visual event localization and video parsing. In ACM Interna- tional Conference on Multimedia , pages 6241–6249, 2022. 1, 3, 6

2022
[42]

X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learn- ing, pages 7164–7173, 2019. 6

2019
[43]

Yung-Hsuan, C

L. Yung-Hsuan, C. Yen-Chun, and F. W. Yu-Chiang. Modality-independent teachers meet weakly-supervised audio-visual event parser. In Neural Information Processing Systems, pages 1–19, 2023. 2, 3, 7

2023
[44]

J. Zhou, D. Guo, and M. Wang. Contrastive positive sam- ple propagation along the audio-visual event line. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(6):7239–7257, 2023. 3, 6, 7

2023
[45]

J. Zhou, D. Guo, Y . Zhong, and M. Wang. Improving audio-visual video parsing with pseudo visual labels. arXiv preprint arXiv:2303.02344, 2023. 1

2023 arXiv
[46]

J. Zhou, D. Guo, Y . Zhong, and M. Wang. Advancing weakly-supervised audio-visual video parsing via segment- wise pseudo labeling. International Journal of Computer Vi- sion, page Early Access, 2024. 3 11

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Arandjelovic and A

R. Arandjelovic and A. Zisserman. Look, listen and learn. In IEEE international conference on computer vision , pages 609–617, 2017. 3

work page 2017

[2] [2]

Aytar, C

Y . Aytar, C. V ondrick, and A. Torralba. Soundnet: Learning sound representations from unlabeled video. Neural Infor- mation Processing Systems, pages 892–900, 2016. 3

work page 2016

[3] [3]

Bayoudh, R

K. Bayoudh, R. Knani, F. Hamdaoui, and A. Mtibaa. A sur- vey on deep multimodal learning for computer vision: ad- vances, trends, applications, and datasets. The Visual Com- puter, pages 1–32, 2021. 1

work page 2021

[4] [4]

D. A. Bulkin and J. M. Groh. Seeing sounds: visual and auditory interactions in the brain. Current Opinion in Neu- robiology, 16(4):415–419, 2006. 1

work page 2006

[5] [5]

H. Chen, D. Zhu, G. Zhang, W. Shi, X. Zhang, and J. Li. Cm- cs: Cross-modal common-specific feature learning for audio- visual video parsing. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 1–5, 2023. 1

work page 2023

[6] [6]

Cheng, Z

H. Cheng, Z. Liu, H. Zhou, C. Qian, W. Wu, and L. Wang. Joint-modal label denoising for weakly-supervised audio- visual video parsing. In European Conference on Computer Vision, pages 431–448. Springer, 2022. 1, 2, 3, 6

work page 2022

[7] [7]

Cheng, R

Y . Cheng, R. Wang, Z. Pan, R. Feng, and Y . Zhang. Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning. In ACM International Conference on Multimedia, pages 3884–3892, 2020. 3

work page 2020

[8] [8]

Ephrat, I

A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein. Look- ing to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. arXiv preprint arXiv:1804.03619, 2018. 3

arXiv 2018