REVIEW 9 cited by
Frame-Voyager: Learning to Query Frames for Video Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.
Forward citations
Cited by 9 Pith papers
-
CREST: Curvature-Regulated Event-Centric Sampling for Efficient Long-Video Understanding
CREST uses local curvature of query-frame relevance over time to select informative frames, outperforming a lightweight baseline and approaching a costly pipeline at far lower preprocessing cost on long-video benchmarks.
-
Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding
EviSelect uses the target multimodal model's internal attention as a prior to dynamically select frames, sampling rates, and resolutions, achieving about 50% token reduction and a 3.9x speedup with better benchmark accuracy.
-
FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding
A training-free frame-selection method that weights frame embeddings by query relevance and maximizes the selected subspace's volume improves keyframe recall and VQA accuracy across eight MLLMs on Video-MME and LongVi...
-
SkyVLaM: Multimodal Large Language Model for UAV Video Understanding in Remote Sensing
SkyVLaM introduces a temporal basis perceiver and adaptive dense selection to improve language-conditioned video segmentation in UAV scenes, and contributes the SkyVid dataset.
-
Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
Attention maps from a small MLLM can serve as a training-free, query-conditioned frame selector, improving long-video QA accuracy under fixed frame budgets.
-
Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding
ReMem improves zero-shot long-video QA by combining LLM-based temporal granularity parsing, CLIP-based dual-semantic frame scoring, and structure-aware dynamic frame routing.
-
Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs
MS-SFNN encodes multi-scale Fourier features in a separable product of fixed-weight cosine subnetworks and solves for linear coefficients by least squares, claiming better accuracy than PINN and SV-SNN on high-frequency PDEs.
-
AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs
AutoV selects instance- and query-specific visual prompts via loss-based pairwise ranking, consistently improving LVLMs across many benchmarks with no backbone fine-tuning.
-
ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning
ViaRL uses rule-based reinforcement learning to train a frame selector for video QA, improving Qwen2.5-VL on VideoMME, LVBench, and MLVU by several points.
Discussion (0). Continue with ORCID to comment.