Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free frame selector that reads the question beats uniform sampling for video LLMs.

desk verdict Q-Frame is a plausible training-free frame-selection plug-in with solid Qwen2-VL gains, but the GPT-4o MLVU result is noise, the LongVideoBench tuning is leaky, and the abstract overclaims on temporal reasoning. read the letter →

arxiv 2506.22139 v3 pith:WUMMBWJR submitted 2025-06-27 cs.CV

Shaojie Zhang , Jiahui Yang , Jianqin Yin , Zhenbo Luo , Jian Luan This is my paper
classification cs.CV
keywords videounderstandingframeselectionGumbel-MaxtrickCLIPretrievalmulti-resolutionadaptationVideo-LLMlongQAquery-awaresampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Q-Frame proposes that a video question can be answered better by choosing frames on the basis of the question itself, rather than taking frames at fixed intervals. The method uses a pretrained text-image matcher to score candidate frames against the query, adds Gumbel noise so the selection explores, and then assigns high, medium, or low resolution by relevance rank so more frames fit the same visual-token budget. The paper reports consistent gains over uniform sampling on MLVU, LongVideoBench, and Video-MME, with Qwen2-VL improving from 56.9 to 65.4 on MLVU, 53.5 to 58.4 on LongVideoBench, and 53.7 to 58.3 on Video-MME without subtitles. The significance, if the claim holds, is a training-free, model-agnostic upgrade that any Video-LLM can adopt.

What carries the argument

Three components carry the argument. Cross-modal Query Retrieval (CQR) embeds candidate frames and the query with a CLIP-like model and computes inner-product similarity scores. Query-aware Frame Selection (QFS) applies a temperature-scaled softmax and the Gumbel-Max trick—adding independent Gumbel noise to the log-probabilities and taking the top K—which turns a deterministic retrieval score into a stochastic sampler without any training. Multi-Resolution Adaptation (MRA) then assigns each selected frame a resolution tier by rank, with the ratio $r(1)=4r(2)=16r(3)$, so token cost falls sharply for low-relevance frames; the resolution constraint $K + M/4 + N/16 = 8$ expresses the fixed budget in units of high-resolution frames. The Gumbel-Max trick is what makes the selection training-free, and the constraint is what lets more frames be presented under the same token limit.

What would settle it

Run Q-Frame on a set of video questions whose answers are only determinable by the order or causality of events across timestamps, such as 'what happens immediately before X', and check whether the frames containing the evidence fall inside the top-K selection; the paper's appendix already shows a water-break timing question where both Q-Frame and uniform sampling fail, and a systematic drop on such a temporal-reasoning subtask would bound the method's scope.

Watch

Extended reading notes

Core claim

The paper's central claim is that the video frames an LLM needs for a question can be recovered from per-frame similarity to the question text, and that a fixed computational budget is best spent by giving the most relevant frames high resolution and less relevant frames low resolution, effectively multiplying how many frames that budget can carry. Q-Frame selects the top-K frames from a Gumbel-perturbed softmax over CLIP scores, then splits the selected frames into three resolution tiers satisfying $r(1)=4r(2)=16r(3)$; with Qwen2-VL the best configuration is 4 high-, 8 medium-, and 32 low-resolution frames, which the token budget treats as equivalent to 8 uniform frames. In the paper's evaluation this yields consistent improvements over uniform sampling across three long-video benchmarks and across open- and closed-source backbones, including GPT-4o. The authors explicitly do not claim the selection preserves temporal order; Q-Frame is positioned as query-aware keyframe retrieval plus resolution allocation.

Load-bearing premise

The method assumes that a per-frame CLIP score against the question, plus random noise, is enough to pick the frames a video model needs, including questions whose answer depends on how events unfold over time rather than on what a single frame shows.

Editorial extensions

If this is right

  • Video-LLM inference can be upgraded without fine-tuning, since any model that accepts visual tokens can ingest the selected frames and per-frame resolutions.
  • Longer videos become more tractable: sampling 128 candidate frames and passing 4 high-, 8 medium-, and 32 low-resolution frames keeps accuracy or improves it while using roughly the token budget of 8 uniform frames.
  • The token budget becomes a resolution-allocation problem, so fine-grained questions about small objects and OCR text benefit from reserving high resolution for the most relevant frames.
  • Closed-source API models also improve, because selection happens outside the model; the paper reports Q-Frame lifts GPT-4o on Video-MME without subtitles from 61.9 to 63.8.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because selection is a per-frame retrieval step, the approach should transfer to other retrieval-style multimodal tasks such as video retrieval or grounded captioning, where the answer signal lives in a few frames.
  • The Gumbel noise is resampled for each query, so drawing several frame subsets and averaging the model's answers could act as an ensemble, a test the paper does not run.
  • The multi-resolution idea could be inverted: tasks that need spatial detail everywhere could trade total frame count for more high-resolution frames, converting the same token budget into coverage or detail as needed.
  • The method's own bad-case analysis predicts a clear boundary: temporal reasoning that requires ordering events is not handled, so any extension that preserves inter-frame order would directly widen the method's scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes Q-Frame, a training-free, plug-and-play frame-selection and multi-resolution adaptation method for Video-LLMs. It uses CLIP-style image-text similarity to score uniformly sampled candidate frames, adds Gumbel noise, selects top-K frames, and allocates different resolutions (high/medium/low) to preserve fine details while fitting a fixed token budget. The method is evaluated on MLVU, LongVideoBench, and Video-MME with VILA-V1.5, GPT-4o, and Qwen2-VL backbones. The headline results show consistent gains for Qwen2-VL (e.g., MLVU 56.9→65.4, LongVideoBench 53.5→58.4, Video-MME without subtitles 53.7→58.3) and for VILA-V1.5 on MLVU and LongVideoBench, while GPT-4o gains are small and the MLVU baseline is unstable across runs. The paper includes ablations on QFS vs. MRA, resolution allocation, temperature, CLIP backbone, and candidate-frame count.

Significance. If the reported gains hold, Q-Frame would be a practically valuable addition to Video-LLM inference: it is model-agnostic, requires no training, and its ablations show that query-relevance-based selection alone improves over uniform sampling at fixed resolution (Table 4). The multi-resolution ablation (Table 5) and the overhead measurements (Table 14) are useful contributions. The paper also provides detailed per-subtask results, which are commendable. However, the central claim that Q-Frame 'preserv[es] critical temporal and spatial information' and 'consistently outperforms uniform frame sampling' is contradicted by the paper's own limitation statement and per-subtask temporal results; the GPT-4o MLVU evidence is within run-to-run noise; and hyperparameters appear to be selected on the same benchmark used for reporting. The method may still be valuable for query-dependent, spatially grounded tasks, but the current claims overreach.

major comments (4)
  1. [Abstract and Appendix B.5; Table 9] The abstract claims Q-Frame 'preserv[es] critical temporal and spatial information' and 'consistently outperforms uniform frame sampling.' Appendix B.5 explicitly concedes that the selection process 'doesn't preserve the sequential structure or causal relationships between events that are critical for effective temporal reasoning.' Table 9 shows multiple negative deltas on temporal subtasks, including VILA-V1.5 dropping from 53.0 to 40.9 on T3O and from 41.1 to 34.2 on T3E, and Qwen2-VL dropping from 78.5 to 70.8 on T2E. This is an internal inconsistency between the stated contribution and the evidence. Please narrow the central claim to tasks where per-frame relevance is sufficient, or provide separate evidence that temporal-reasoning performance is preserved (e.g., with temporal-aware selection).
  2. [Appendix B.2 and Table 2] The GPT-4o MLVU result is not supported by the data. Appendix B.2 reports three baseline runs of 27.4, 28.6, and 28.3, and a Q-Frame run of 29.3; the manuscript reports 28.6 as the baseline, yielding a 0.7-point gap that is smaller than the baseline's own run-to-run spread of 1.2 points. This difference is within noise and cannot support the claim that Q-Frame improves GPT-4o on MLVU. Please either provide statistically meaningful evidence (multiple Q-Frame runs with error bars and a significance test) or remove/reframe this specific claim.
  3. [Section 4.3 and Appendix B.4] Hyperparameters including temperature tau (Table 13), resolution allocation K/M/N (Table 6), the CLIP-like backbone (Table 12), and candidate-frame count T (Table 14) are tuned by peeking at LongVideoBench and then reported on the same benchmark. Since no held-out validation is described, the reported gains may be inflated by selection on the test set. Please evaluate the method on a validation split or, failing that, report sensitivity across a range of hyperparameters and state explicitly that the reported configuration was chosen on the benchmark itself; the 'consistently outperforms' claim should then be tempered accordingly.
  4. [Section 3.4, Eq. (8)] The token-budget equation K + M/4 + N/16 = 8 assumes that visual-token consumption scales exactly with the linear resolution ratio r(1):r(2):r(3) = 1:2:4. Table 6 shows that actual tokens/video vary from 2265.1 to 2370.1, a 4.6% range, so the equation is at best approximate. More importantly, Qwen2-VL's native dynamic tokenization may not follow a simple area-law scaling. Please report actual token counts and inference latency for the configurations in Table 6, and justify that the approximation is tight enough to support the claim that Q-Frame 'process more frames without exceeding computational limits.'
minor comments (3)
  1. [Appendix B.2] The table numbering is inconsistent: the repeated GPT-4o experiment is referred to as 'Table 10' in the text, but the presented tables are labeled Table 8 (parameter config), Table 8 (repeated runs), Table 9 (subtasks), and Table 10 (MLVU subtasks). Please renumber consistently and fix the in-text references.
  2. [Table 3] The duration ranges '(8s, 15s]' and '(15s, 1m]' skip the interval (1m, 3m]; please clarify whether the (1m, 3m] category was omitted or merged.
  3. [Figure 4] The bar chart for Video-MME task accuracies is difficult to read because the baseline and Q-Frame values are close and the y-axis labels are not visible. Please add value labels and a legend.

Circularity Check

1 steps flagged · score 6.0 of 10

LongVideoBench headline result is the argmax of hyperparameter ablations run on the same benchmark; MLVU and Video-MME results remain independent.

  1. fitted input called prediction [Section 4.3, 'Frames’ resolution allocation' (Table 6) and Section 4.2.1 (Table 2)]
    "The optimal configuration includes 4 high-resolution frames, 8 medium-resolution frames, and 32 low-resolution frames, achieving an accuracy of 58.4%."

    LongVideoBench serves both as the tuning set and as a reported benchmark. Section 4.3 states that the ablation studies are 'on LongVideoBench using Qwen2-VL as the backbone,' and Table 6 selects K=4, M=8, N=32 because that configuration achieves 58.4%. The headline Table 2 then reports Qwen2-VL + Q-Frame as 58.4 on LongVideoBench. Additionally, Table 13 selects tau by LongVideoBench accuracy and Table 12 selects the CLIP-like backbone by LongVideoBench accuracy. Thus the LongVideoBench improvement over uniform sampling is the optimum of the selection procedure on that same dataset, not an out-of-sample prediction, so the 'consistently outperforms uniform frame sampling' claim is partially forced for this benchmark.

full rationale

Q-Frame's mechanism is a standard CLIP-based similarity score, a Gumbel-Max sampler, and a resolution re-allocation; there is no derivation chain in which the output is defined by the method, and no load-bearing self-citation chain is present. The Gumbel-Max trick and the external benchmarks are independent support. The one significant circularity is evaluative: the K/M/N allocation, temperature tau, and CLIP backbone are all chosen by maximizing accuracy on LongVideoBench, and the same LongVideoBench value is then presented as a headline result. That particular prediction reduces to the selected optimum of the tuning procedure. The MLVU and Video-MME numbers were not used for selection, so they retain genuine predictive content. Appendix B.5's concession that Q-Frame 'doesn't preserve the sequential structure or causal relationships between events' weakens the broad temporal-reasoning claim, but that is an internal-consistency issue rather than a circular derivation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. It relies on standard CLIP embeddings and standard Gumbel-Max sampling. The cost is paid in assumptions and hyperparameters: five hyperparameter choices are tuned, largely on LongVideoBench, and the temporal-coverage assumption is explicitly acknowledged as a limitation. These should count against the paper's contribution ledger.

free parameters (5)
  • temperature tau = 0.8 (Table 13; text erroneously says 1.2)
    Controls softmax sharpness in Eq. 3 before Gumbel-Max sampling; tuned on LongVideoBench and reported as an ablation.
  • resolution allocation K, M, N = 4, 8, 32
    Number of high, medium, and low resolution frames; chosen by ablation on LongVideoBench (Table 6) and satisfies Eq. 8.
  • candidate frame count T = 128
    Number of uniformly sampled candidate frames; set to match Frame-Voyager; ablation in Table 14 shows stable results.
  • CLIP-like backbone choice = Long-CLIP
    Selected among CLIP, SigLIP, and Long-CLIP based on LongVideoBench accuracy (Table 12).
  • resolution ratio r(1):r(2):r(3) = area ratio 1:4:16 (low:medium:high)
    Defines token equivalence in Eq. 8; the paper writes r(1)=4r(2)=16r(3), interpreted as area scaling; used to equalize token budgets.
assumptions (4)
  • standard math Gumbel-Max trick produces valid samples from the softmax distribution over CLIP scores.
    Used in Section 3.3 (Eqs. 3, 4, 5) to justify stochastic top-K frame selection.
  • domain assumption CLIP-like image-text embeddings reliably measure query relevance of individual video frames for Video-LLM QA.
    Section 3.2 states pre-trained VLMs offer 'excellent zero-shot capabilities for cross-modal retrieval'; all frame selection depends on this.
  • ad hoc to paper Frame-level relevance plus Gumbel noise is sufficient to preserve temporal information needed for the tasks.
    Section 3.3 admits CLIP neglects temporal correlation and adds noise as the only remedy; Appendix B.5 concedes temporal reasoning tasks fail because causal order is lost.
  • ad hoc to paper Eq. 8 (K + M/4 + N/16 = 8) approximates actual visual token consumption across resolutions.
    Used to claim a fixed token budget; Table 6 shows the chosen configuration actually uses +3.5% more tokens than the 8-frame baseline, so the equality is approximate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs." pith.science (2026). https://pith.science/paper/WUMMBWJR

@misc{pith2026250622139,
  author       = {Pith},
  title        = {Pith review of: Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WUMMBWJR}},
  note         = {Machine review of arXiv:2506.22139}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated significant success in visual understanding tasks. However, challenges persist in adapting these models for video comprehension due to the large volume of data and temporal complexity. Existing Video-LLMs using uniform frame sampling often struggle to capture the query-related crucial spatiotemporal clues of videos effectively. In this paper, we introduce Q-Frame, a novel approach for adaptive frame selection and multi-resolution scaling tailored to the video's content and the specific query. Q-Frame employs a training-free, plug-and-play strategy generated by a text-image matching network like CLIP, utilizing the Gumbel-Max trick for efficient frame selection. Q-Frame allows Video-LLMs to process more frames without exceeding computational limits, thereby preserving critical temporal and spatial information. We demonstrate Q-Frame's effectiveness through extensive experiments on benchmark datasets, including MLVU, LongVideoBench, and Video-MME, illustrating its superiority over existing methods and its applicability across various video understanding tasks.

Figures

Figures reproduced from arXiv: 2506.22139 by the authors.

Figure 1
Figure 1. Comparison of uniform frame sampling and proposed [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall accuracy (%) Video-MME (without subtitles) for [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The overall framework of Q-Frame. Q-Frame is composed of Cross-modal Query Retrieval (CQR), Query-Aware Frame Selection [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Accuracies (%) of Qwen2-VL-Video, Qwen2-VL, and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Case analysis from Video-MME [7]. Uniform sampling captures only a limited number of frames relevant to the query. While our Q-Frame extracts more relevant frames with a variety of resolutions. 4.4. Qualitative Analysis [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Case analysis from Video-MME. : uniform sampling; : Q-Frame. Video-LLMs with Q-Frame can capture the keyframes of the car and provide answers correctly [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Case analysis from Video-MME. : uniform sampling; : Q-Frame. Video-LLMs with Q-Frame can capture the keyframes of the action of dunking and respond accordingly [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Case analysis from Video-MME. : uniform sampling; : Q-Frame. Video-LLMs with Q-Frame can capture the keyframes of the young girl and the elderly man and respond accordingly [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Case analysis from Video-MME. : uniform sampling; : Q-Frame. Neither our Q-Frame nor the uniform sampling can correctly handle such a temporal reasoning task [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    MS-SFNN encodes multi-scale Fourier features in a separable product of fixed-weight cosine subnetworks and solves for linear coefficients by least squares, claiming better accuracy than PINN and SV-SNN on high-frequency PDEs.

Reference graph

Works this paper leans on

42 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 1, 5, 6

    arXiv 2023

  2. [2]

    Robust motion-guided frame sam- pler with interpretive evaluation for video action recognition

    Jing Bai, Yuxiang Zhang, Yiran Wang, Zhu Xiao, Yong Xiong, and Licheng Jiao. Robust motion-guided frame sam- pler with interpretive evaluation for video action recognition. IEEE Transactions on Mobile Computing, 2025. 3

    work page 2025

  3. [3]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. 37:19472–19495, 2025. 3

    work page 2025

  4. [4]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 3

    2024

  5. [5]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1, 2, 6

    arXiv 2024

  6. [6]

    Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023, 2024. 3, 6

    arXiv 2024

  7. [7]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 2, 5, 8, 11

    arXiv 2024

  8. [8]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. 2, 4

    arXiv 2016

Show all 42 references
  1. [9]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  2. [10]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 6

    2024 arXiv

  3. [11]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  4. [12]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  5. [13]

    Keyvideollm: Towards large-scale video keyframe selection

    Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang, Conghui He, Bin Cui, Chong Chen, and Wentao Zhang. Keyvideollm: Towards large-scale video keyframe selection. arXiv preprint arXiv:2407.03104, 2024. 2, 3

    2024 arXiv

  6. [14]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. 2024. 1, 2, 3, 4, 6

    2024

  7. [15]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 2, 5, 6, 7, 12

    2024

  8. [16]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023. 3

    2023

  9. [17]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 3

    2024

  10. [18]

    Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024. 3, 6

    2024

  11. [19]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xi- aoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542,

    arXiv

  12. [20]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Pro- ceedings of the Annual Meeting of the Association for Com- putational Linguistics, 2024. 1, 2, 3, 4

    2024

  13. [21]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. 1, 2, 5, 6, 7, 11, 12

    2024

  14. [22]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In Proceedings of the International Conference ...

    2021

  15. [23]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14313–14323, 2024. 3

    2024

  16. [24]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14485, 2024. 6

    2024 arXiv

  17. [25]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  18. [26]

    Video understanding with large language models: A survey

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432, 2023. 2

    2023

  19. [27]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 2, 3, 5, 6, 7, 12

    2024 arXiv

  20. [28]

    Internvideo2: Scaling foundation models for multi- modal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multi- modal video understanding. In Proceedings of the European Conference on Computer Vision, pages 396–416, 2...

    2024

  21. [29]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. 37:28828–28857, 2025. 2, 5, 11

    2025

  22. [30]

    Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition

    Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen. Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6222–6231, 2019. 3

    2019

  23. [31]

    Adaframe: Adaptive frame selection for fast video recognition

    Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry S Davis. Adaframe: Adaptive frame selection for fast video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1278–1287, 2019. 3

    2019

  24. [32]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 3, 6

    2024 arXiv

  25. [33]

    Frame-voyager: Learning to query frames for video large language models

    Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, et al. Frame-voyager: Learning to query frames for video large language models. 2025. 2, 3, 5, 6, 13

    2025

  26. [34]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 13

    2023

  27. [35]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 3

    2023 arXiv

  28. [36]

    Lmms-eval: Re- ality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Re- ality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024. 5

    2024 arXiv

  29. [37]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 3, 6, 12, 13

    2024 arXiv

  30. [38]

    Direct preference optimiza- tion of video large multimodal models from language model reward

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimiza- tion of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258, 2024. 3

    2024 arXiv

  31. [39]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 2, 3, 4

    2024 arXiv

  32. [40]

    Mgsampler: An explainable sampling strategy for video ac- tion recognition

    Yuan Zhi, Zhan Tong, Limin Wang, and Gangshan Wu. Mgsampler: An explainable sampling strategy for video ac- tion recognition. In Proceedings of the IEEE/CVF Inter- national conference on Computer Vision, pages 1513–1522,

  33. [41]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  34. [2024]

    Limitations Q-Frame enhances query-aware video understanding, but it depends on pre-trained models, lacks explicit temporal modeling, and operates within a fixed token budget

    2, 5, 11 Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs Supplementary Material A. Limitations Q-Frame enhances query-aware video understanding, but it depends on pre-trained models, lacks explicit temporal modeling, and operates within a fi...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.