REVIEW 6 cited by
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.
Forward citations
Cited by 6 Pith papers
-
HIPPO-Video: Simulating Watch Histories with Large Language Models for Personalized Video Highlighting
HIPPO-Video contributes 2,040 LLM-simulated watch-history and saliency-score pairs, and the HiPHer model uses these histories to beat generic and query-based baselines on the new benchmark.
-
NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding
NarrativeTrack shows that video AI models, including GPT-4o, falter at tracking a specific person across scene changes, outfit changes, and similar-looking characters in long videos.
-
From Coarse to Nuanced: Cross-Modal Alignment of Fine-Grained Linguistic Cues and Visual Salient Regions for Dynamic Emotion Recognition
GRACE pairs motion-weighted video tokens with AI-refined emotion text tokens using optimal transport, reporting new UAR and WAR records on DFEW, FERV39k, and MAFW.
-
Video-Text Temporal Localization via Multi-Scale Convolution and Dynamic Routing
Multi-scale temporal convolutions plus capsule routing improve video-text moment localization to 42.9% R@0.5 and 41.1% mIoU on ActivityNet Captions.
-
THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage
THYME fuses the authors' prior HIG hierarchy and CYCLO cyclic attention into one model, reporting modest recall gains on the ASPIRe benchmark and on its own new AeroEye-v1.0 dataset.
-
Generalizing vision-language models to novel domains: A comprehensive survey
A survey of VLM generalization literature organized by transferred module, with benchmark tables and a review of multimodal LLMs.
Discussion (0). Sign in to comment.