Temporal-Spatial Feature Pyramid for Video Saliency Detection
read the original abstract
Multi-level features are important for saliency detection. Better combination and use of multi-level features with time information can greatly improve the accuracy of the video saliency model. In order to fully combine multi-level features and make it serve the video saliency model, we propose a 3D fully convolutional encoder-decoder architecture for video saliency detection, which combines scale, space and time information for video saliency modeling. The encoder extracts multi-scale temporal-spatial features from the input continuous video frames, and then constructs temporal-spatial feature pyramid through temporal-spatial convolution and top-down feature integration. The decoder performs hierarchical decoding of temporal-spatial features from different scales, and finally produces a saliency map from the integration of multiple video frames. Our model is simple yet effective, and can run in real time. We perform abundant experiments, and the results indicate that the well-designed structure can improve the precision of video saliency detection significantly. Experimental results on three purely visual video saliency benchmarks and six audio-video saliency benchmarks demonstrate that our method outperforms the existing state-of-the-art methods.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
Exploring deep learning for Event-Based Saliency Prediction with a Transformer-based model
SEST is the first deep learning model for event-based saliency prediction, using a pretrained Swin Transformer backbone and synthetic benchmarks to outperform prior event methods while transferring to real event streams.
-
Attend to Anything: Foundation Model for Unified Human Attention Modeling
AAM unifies attention modeling across modalities and tasks by reformulating attention as hierarchical cognitive entailment via language prompts in hyperbolic space and video attention as Fokker-Planck diffusive evolut...
-
BIAS: A Biologically Inspired Algorithm for Video Saliency Detection
BIAS is a biologically inspired video saliency model that integrates static and motion features via retina-like detection and multi-Gaussian fitting, outperforming baselines on DHF1K and anticipating traffic accidents...
-
ViASNet: A Video Ad Saliency Network for Predicting Dynamic Saliency and Viewer Engagement
ViASNet applies a 3D U-Net architecture augmented with audio and semantic inputs to predict dynamic saliency in video ads and uses frame-wise entropy to diagnose low-engagement scenes on eye-tracked data from 151 ads.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.