Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization

Andrew Gilbert; Edward Fish; Jon Weinbren

Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.03456 v1 pith:ZA4SA3DV submitted 2023-10-05 cs.CV cs.LGcs.MM

Edward Fish , Jon Weinbren , Andrew Gilbert This is my paper · ORCID

classification cs.CVcs.LGcs.MM

keywords featuretemporalaudioaudio-visualactiondatafusionlocalization

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Temporal Action Localization (TAL) aims to identify actions' start, end, and class labels in untrimmed videos. While recent advancements using transformer networks and Feature Pyramid Networks (FPN) have enhanced visual feature recognition in TAL tasks, less progress has been made in the integration of audio features into such frameworks. This paper introduces the Multi-Resolution Audio-Visual Feature Fusion (MRAV-FF), an innovative method to merge audio-visual data across different temporal resolutions. Central to our approach is a hierarchical gated cross-attention mechanism, which discerningly weighs the importance of audio information at diverse temporal scales. Such a technique not only refines the precision of regression boundaries but also bolsters classification confidence. Importantly, MRAV-FF is versatile, making it compatible with existing FPN TAL architectures and offering a significant enhancement in performance when audio data is available.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding
cs.CV 2025-06 conditional novelty 5.0 of 10

DEL is a new audio-visual transformer framework that reports state-of-the-art temporal action localization on UnAV-100, THUMOS14, ActivityNet 1.3, and EPIC-Kitchens-100.

Pith tools