REVIEW 20 cited by
ActionCLIP: A New Paradigm for Video Action Recognition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The canonical approach to video action recognition dictates a neural model to do a classic and standard 1-of-N majority vote task. They are trained to predict a fixed set of predefined categories, limiting their transferable ability on new datasets with unseen concepts. In this paper, we provide a new perspective on action recognition by attaching importance to the semantic information of label texts rather than simply mapping them into numbers. Specifically, we model this task as a video-text matching problem within a multimodal learning framework, which strengthens the video representation with more semantic language supervision and enables our model to do zero-shot action recognition without any further labeled data or parameters requirements. Moreover, to handle the deficiency of label texts and make use of tremendous web data, we propose a new paradigm based on this multimodal learning framework for action recognition, which we dub "pre-train, prompt and fine-tune". This paradigm first learns powerful representations from pre-training on a large amount of web image-text or video-text data. Then it makes the action recognition task to act more like pre-training problems via prompt engineering. Finally, it end-to-end fine-tunes on target datasets to obtain strong performance. We give an instantiation of the new paradigm, ActionCLIP, which not only has superior and flexible zero-shot/few-shot transfer ability but also reaches a top performance on general action recognition task, achieving 83.8% top-1 accuracy on Kinetics-400 with a ViT-B/16 as the backbone. Code is available at https://github.com/sallymmx/ActionCLIP.git
Forward citations
Cited by 20 Pith papers
-
Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment
Peak-End-Net uses image-aesthetic priors and peak-end-rule frame weighting to set a new state of the art on VADB and zero-shot DIVIDE-3K video aesthetic assessment.
-
Breaking the 15% Barrier: A Real-World Data-Driven System for Proactive Social Robot Triggered by User Nonverbal Cues
About 15% of retail robot turns are nonverbal-triggered; a video multi-label recognizer plus LLM token conditioning can produce proactive responses for frequent social cues.
-
What Can We Learn from Harry Potter? An Exploratory Study of Visual Representation Learning from Atypical Videos
Training with atypical videos, such as sci-fi, animation, and accidental actions, improves open-world video tasks more than typical auxiliary data.
-
Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models
Richer text prompts from LLM synonyms and cleaner image regions from activation maps improve zero-shot vision-language classification.
-
Cross-Modal Dual-Causal Learning for Long-Term Action Recognition
CMDCL debiases text embeddings by back-door adjustment and deconfounds video features by front-door adjustment, achieving state-of-the-art long-term action recognition on three benchmarks.
-
"Before, I Asked My Mom, Now I Ask ChatGPT": Visual Privacy Management with Generative AI for Blind and Low-Vision People
Blind and low vision people already use generative AI to protect their visual privacy, and they want future tools to process data locally with zero-retention guarantees and sensitive-content redaction.
-
MReg: A Novel Regression Model with MoE-based Video Feature Mining for Mitral Regurgitation Diagnosis
A regression-based deep learning model with feature selection, amplification, and Mixture-of-Experts pooling grades mitral regurgitation from 4-chamber color Doppler videos with 89.36% accuracy.
-
LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matching
Injecting LLM-generated action triplets and action state descriptions as learnable visual prompts improves CLIP's image-text retrieval performance on Flickr30K and COCO.
-
Feature Hallucination for Self-supervised Action Recognition
New object-detection and saliency descriptors, combined with uncertainty-weighted feature hallucination, improve RGB-only action recognition on multiple video benchmarks.
-
FRAME: Pre-Training Video Feature Representations via Anticipation and Memory
FRAME distills DINO and CLIP features into a compact video encoder with a memory module and future-frame prediction, outperforming image-based and self-supervised video baselines on dense video tasks.
-
Knowledge-guided Disentanglement with Atomic Actions for Action Recognition
LLM-generated atomic action descriptions injected into scene-graph node features improve prompt-guided disentanglement for multi-label action recognition, with reported gains on Charades-oracle and SportsHHI but not o...
-
GHR-VLM: Making Zero-Shot Transit Video Analytics Realizable with Grounded Hybrid Reasoning
A grounded edge-cloud pipeline applies VLM reasoning only to localized passenger and farebox evidence, achieving 31–54% five-class zero-shot payment accuracy on two real bus videos.
-
Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation
A trajectory-conditioned retrieval system discovers multiple motion descriptions in videos without user queries and grounds them to point tracks, evaluated mainly on MeViS.
-
Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space
A four-modality contrastive model with token-level alignment improves text, video, and newly introduced audio-to-motion retrieval on HumanML3D and KIT-ML.
-
Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives
TimesCLIP aligns image-based and text-based views of the same time series via contrastive learning to improve forecasting accuracy on several benchmarks, but the full multimodal model is not used on two of the six lon...
-
MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
MoMa adapts frozen CLIP to video by injecting Mamba-computed scale and bias into each layer, improving accuracy and efficiency on multiple action recognition benchmarks.
-
Video Understanding by Design: How Datasets Shape Video Models
A dataset-centric framework that explains video architectures as responses to structural properties of benchmark datasets.
-
MoExDA: Domain Adaptation for Edge-based Action Recognition
Exchanging feature statistics between RGB and edge streams in a two-stream action recognition ViT reduces background bias while keeping accuracy close to an RGB-only baseline.
-
From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Control
A new 124K-clip dataset with hierarchical text annotations, plus a pipeline that couples an LLM planner, a text-to-pose VAE, diffusion in-betweening, and physics control to generate long-horizon human behaviors.
-
From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
OvSGTR jointly predicts unseen objects and relationships in scene graphs using a DETR-like transformer, relation-aware pre-training, and knowledge distillation, achieving state-of-the-art results on VG150 and GQA200.
Discussion (0). Continue with ORCID to comment.