REVIEW 15 cited by
SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video large language model (LLM) that can jointly capture detailed spatial semantics and long-range temporal context without exceeding the token budget of commonly used LLMs. This is realized by using a two-stream SlowFast design of inputs for Video LLMs to aggregate features from sampled frames in an effective way. Specifically, the Slow pathway extracts features at a low frame rate while keeping as much spatial detail as possible (e.g., with 12x24 tokens), and the Fast pathway operates on a high frame rate but uses a larger spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As a result, this design allows us to adequately capture both spatial and temporal features that are beneficial for detailed video understanding. Experimental results show that SF-LLaVA outperforms existing training-free methods on a wide range of video tasks. On some benchmarks, it achieves comparable or even better performance compared to state-of-the-art Video LLMs that are fine-tuned on video datasets. Code has been made available at: https://github.com/apple/ml-slowfast-llava.
Forward citations
Cited by 15 Pith papers
-
TrajTok: Learning Trajectory Tokens enables better Video Understanding
TrajTok learns to tokenize video into object-trajectory tokens end-to-end, improving video CLIP, probing, and VLM performance over patch and token-merging baselines.
-
SkyVLaM: Multimodal Large Language Model for UAV Video Understanding in Remote Sensing
SkyVLaM introduces a temporal basis perceiver and adaptive dense selection to improve language-conditioned video segmentation in UAV scenes, and contributes the SkyVid dataset.
-
Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
Attention maps from a small MLLM can serve as a training-free, query-conditioned frame selector, improving long-video QA accuracy under fixed frame budgets.
-
Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data
Adding Strefer's synthetic space-time reference questions to video instruction tuning improves mask-referred description/QA, timestamp QA, and temporal reasoning over a video-LLM baseline.
-
IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes
A new multi-shot video dataset and an instance-prompt video LLM report large gains, but the main benchmark is built by the same authors and the model is not released.
-
Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders
Nar-KFC improves long-video question answering by interleaving query-relevant diverse keyframes with captions of non-keyframes, raising accuracy on Video-MME, LongVideoBench, and MLVU across 7B MLLMs.
-
Clapper: Compact Learning and Video Representation in VLMs
Clapper achieves 13x visual token compression in video VLMs with maintained or improved QA accuracy using a slow-fast representation and a TimePerceiver module.
-
UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation
UniGen shows a 1.5B model trained on open data can beat larger systems on image understanding and generation once it verifies its own outputs with chain-of-thought and Best-of-N selection.
-
Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
Video LLM benchmark scores are inflated by language-prior and static-frame questions; VBenchComp separates those from true temporal questions and shows a trimmed core subset preserves rankings.
-
Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.
-
IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning
IntentVCNet uses per-frame object coordinates, red-box visual prompts, and a lightweight box adapter to make video captioning focus on a user-selected object, reporting 225.19 CIDEr on the IntentVC public test set.
-
MANTA: Cross-Modal Semantic Alignment and Information-Theoretic Optimization for Long-form Multimodal Understanding
A multimodal retrieval pipeline that projects video and audio into text and claims near-optimal context selection, with reported gains of up to 22.6% on Video-MME that rest on circular theory and unreleased data.
-
MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows
MOOSE fuses frozen DINOv2 image features with optical flow features via cross-attention and causal aggregation, reporting 70.84% Kinetics-400 and 65.23% SSv2 top-1 accuracy, alongside interpretable attention maps.
-
APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention
SPAVA compresses each GPU's video context into essential key-values for sharing, making multi-GPU long-video prefilling 1.18x faster than APB and 12.72x faster than single-GPU FlashAttention with near-equal accuracy.
-
Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation
MedRegion-CT integrates region-representative tokens, mask-driven segmentation tokens, and patient-specific attribute prompts into a multimodal LLM, reporting state-of-the-art scores on RadGenome-Chest CT report generation.
Discussion (0). Continue with ORCID to comment.