Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that multiple-choice long-video benchmarks inflate AI performance and that open-ended short-answer questions reveal a much lower, more realistic capability; it presents VideoEval-Pro as that more faithful measure.

desk verdict Real and useful observation about MCQ inflation, but the benchmark's validity claims outrun the evidence—the paper's own Figure 4 shows a prior-answerable question survived filtering. read the letter →

arxiv 2505.14640 v1 pith:QFQWN44T submitted 2025-05-20 cs.CV

Wentao Ma , Weiming Ren , Yiming Jia , Zhuofeng Li , Ping Nie , Ge Zhang , Wenhu Chen This is my paper · ORCID
classification cs.CV
keywords longvideounderstandingopen-endedQAmultiple-choiceinflationbenchmarkfilteringLLM-as-a-judgeframescalinglargemultimodalmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VideoEval-Pro claims that current long-video benchmarks are inflated: multiple-choice questions let models guess, and many questions can be answered from priors with a single random frame, so reported accuracies do not reflect true video understanding. The paper's answer is a new benchmark of 1,289 open-ended short-answer questions over 465 videos averaging 38 minutes, built by converting MCQs from four existing benchmarks into free-form questions with the correct option as the gold answer and then filtering out questions answerable without video. Evaluated across 21 proprietary and open-source video models, open-ended accuracy drops by more than 25 percentage points from MCQ accuracy, MCQ scores fail to predict open-ended scores, and accuracy improves steadily when more input frames are given, unlike existing benchmarks. The best model, GPT-4.1, reaches only 40.8% open-ended accuracy, which the paper presents as a clearer picture of how far long-video understanding still has to go.

What carries the argument

The load-bearing mechanism is the MCQ-to-open conversion followed by leakage filtering. Each multiple-choice question becomes a short-answer question by keeping the question stem and the correct option as the gold answer and discarding the distractors, so the model cannot exploit option pattern-matching. The filtering pipeline removes videos under 10 minutes, questions whose options average more than five words, questions judged unanswerable from video content alone by Gemini-2.0-Flash, and questions that the same model can answer correctly from one randomly sampled frame. Evaluation then uses GPT-4o-0806 as an LLM judge with the SimpleQA rubric of correct, incorrect, and not-attempted, and frame-scaling curves track accuracy at 1, 32, 64, 128, 256, and 512 sampled frames. The pipeline is what converts a collection of existing MCQs into a benchmark the paper argues cannot be gamed by priors or single-frame guessing.

What would settle it

Give a random sample of VideoEval-Pro questions to human raters who never see the videos, with only the question and no options, and require a short answer; if their match rate against the gold answers is well above zero, the claim that retained questions cannot be answered from priors fails. A per-item version is to check each gold answer against the question text for world-knowledge cues, starting with the paper's Figure 4 Toronto memorial example.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that long-video understanding is substantially weaker than MCQ benchmarks suggest, and that a benchmark can be made more faithful by reformulating questions so that the model must produce a short fact rather than choose among options. It reports that for the same questions, all 21 evaluated models drop by at least roughly 16 points and often more than 25 points when options are removed, and that model rankings change: models that win on MCQs, such as InternVL2.5 and InternVL3, lose to Qwen2.5-VL on open-ended answers. It further shows that one input frame already gives around 45 to 50 percent accuracy on Video-MME, whereas one frame gives only about 10 percent on VideoEval-Pro, and that VideoEval-Pro accuracy rises consistently as frames increase from 1 to 512, whereas Video-MME accuracy saturates or falls at 512 frames. These contrasts are the evidence that open-ended short-answer evaluation is the more realistic measure.

Load-bearing premise

The benchmark's validity rests on the assumption that converting a multiple-choice question into a free-form one by keeping only the correct option produces an answer that is uniquely determined by the video, and that the one-frame filtering step removes every question answerable from general knowledge; the paper's own Toronto war memorial example shows this assumption failing for at least one retained question.

Editorial extensions

If this is right

  • MCQ accuracy on long-video benchmarks should not be read as true long-video capability; rankings based on MCQs can invert when the same questions are asked open-ended.
  • Frame count is a meaningful evaluation axis: VideoEval-Pro rewards more frames with steady gains, so models that plateau with added frames are likely exploiting priors rather than integrating video content.
  • The reported gap between proprietary and open-source models reverses on VideoEval-Pro, suggesting that open-source models' strong MCQ results are brittle rather than evidence of comparable long-video understanding.
  • At 40.8% open-ended accuracy for the best model, the benchmark indicates that long-video understanding remains far from solved and leaves clear headroom for model improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A per-item audit of the released questions is needed: if any gold answer is common knowledge rather than video-derived, the filtering step has not established the benchmark's grounding, and the paper's own Remembrance War Memorial example shows this failure mode.
  • The MCQ-to-open gap may partly measure answer-generation and format-handling skill, not only video understanding; a model that knows the right fact can still fail to phrase it within the judge's rubric, so open-ended scores may understate as well as MCQ scores overstate.
  • A natural extension is to score the same items with human annotators under three conditions: no video, one frame, and full video; that calibration would quantify how much of VideoEval-Pro's difficulty is video-dependent.
  • Because the difficulty filter uses one random frame from one model, other models with stronger priors or different sampling could still answer some retained items without video; the benchmark's leakage resistance is bounded by the filter model's knowledge.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that existing long-video understanding benchmarks are inflated and unreliable because they rely on multiple-choice questions (MCQs) that allow guessing and on questions answerable from strong priors. It proposes VideoEval-Pro, a benchmark that converts MCQ questions from Video-MME, MLVU, LVBench, and LongVideoBench into open-ended short-answer questions, then filters them by video duration, answer/option length, answerability, and difficulty, yielding 1,289 questions over 465 videos. The paper evaluates 21 proprietary and open-source LMMs and reports three main findings: open-ended accuracy drops by more than 25% compared to MCQ accuracy, MCQ and open-ended scores are not strongly correlated, and performance on VideoEval-Pro improves monotonically with more input frames, unlike existing benchmarks. The paper concludes that VideoEval-Pro provides a more realistic and reliable measure of long video understanding.

Significance. If the filtering pipeline were validated, VideoEval-Pro would address a real and timely problem: MCQ-based LVU scores are plausibly inflated by option artifacts and world-knowledge priors. The paper's strengths include broad model coverage (21 models), a direct MCQ-versus-open comparison on identical questions, a publicly released dataset, and transparent qualitative examples. However, the central validity claim rests on filtering guarantees that the paper's own Figure 4 contradicts, so the benchmark is promising but not yet established as a reliable measure of long-video understanding.

major comments (4)
  1. [§3.1 (Answerability and Difficulty Filtering), Figure 4] The load-bearing premise that every retained question is answerable only from video content is not established. The conversion step in §3.1 makes the correct MCQ option the gold answer, and the filters are single-model proxies: Gemini-2.0-Flash answers from a random frame, and the answerability prompt asks the model to self-assess. Figure 4 shows a retained question, "What is at the Remembrance War Memorial in Toronto, Canada?" with gold "Thousands of Canadian flags," for which the filter model itself responded that it cannot be answered from the provided video. This is direct evidence that prior-answerable questions can survive the pipeline, so the reported >25% MCQ-to-open drop and the low single-frame accuracy are confounded with world-knowledge leakage. I request a no-video baseline (e.g., GPT-4o and Gemini-1.5-Pro answering without any video frames) and a human audit of the 1,289 gold answers to quantify the leak.
  2. [§3.1 (Difficulty Filtering) and §4.3, Figure 3a] The frame-scaling finding is partly circular. The difficulty filter removes any question that Gemini-2.0-Flash can answer correctly from a single random frame; consequently, low accuracy at 1 frame and monotone improvement with more frames are properties enforced by construction rather than independent discoveries about the benchmark. To support the claim that VideoEval-Pro "requires rich temporal information," the authors should either report the frame-scaling curve before filtering or compare retained versus removed questions. As written, Figure 3a cannot by itself distinguish temporal understanding from a floor effect induced by the filter.
  3. [§3.3 and Appendix E] All open-ended accuracy numbers, including the headline drops, depend on GPT-4o-0806 as judge, yet no validation of the judge is reported. The SimpleQA-style rubric classifies hedged or partially correct answers as Correct or Not Attempted, and small differences in judging can move scores by several points; the paper presents no human-LLM agreement rate on VideoEval-Pro. I recommend reporting agreement on a sample of at least 100 responses and the judge's sensitivity to rubric variations (e.g., treating "Not Attempted" as incorrect).
  4. [§4.2, Table 2] The claim that "higher MCQ scores do not lead to higher open-ended scores" is supported only by informal pair comparisons (e.g., InternVL2.5/InternVL3 vs. Qwen2.5-VL) rather than a correlation or rank test. With 21 models and no error bars, the apparent dissociation between MCQ and open-ended ranks may not be statistically reliable. Please report Spearman or Pearson correlations between the MCQ and Open columns, or per-model bootstrap intervals, to substantiate the conclusion.
minor comments (6)
  1. [§3.1] The "average word count of answer options exceeds five words" threshold is stated without justification; please provide the distribution of option lengths or a sensitivity analysis.
  2. [§3.1] The random single-frame sampling has no seed; include the seed or the frame index protocol for reproducibility.
  3. [§4.3, Figures 1 and 3] The captions of Figures 1 and 3 do not fully explain the MCQ accuracy curves on Video-MME; specify the evaluation metric and confirm that the same frame settings are used for both benchmarks.
  4. [Appendix E] The judge prompt contains a numbering error ("Predicted answer 4" appears twice) and a formatting typo in the numeric example; these should be corrected in the released prompt.
  5. [§3.3] The cross-reference "as shown in Appendix 5" should point to the actual appendix section letter (Appendix E).
  6. [Table 3] The row label "∆(Open - Proprietary)" is confusing; relabel it as "Open-source minus Proprietary" and add a note that negative values indicate that proprietary models perform better on VideoEval-Pro.

Circularity Check

1 steps flagged · score 6.0 of 10

The frame-scaling and single-frame-resistance findings are enforced by the construction pipeline rather than discovered from model behavior.

  1. self definitional [Section 3.1 (Difficulty Filtering) / Section 4.3 (Frame Scaling Properties)]
    "we filter out questions that are too easy to answer. To identify such cases, we randomly sample a single frame from each input video and prompt Gemini-2.0-Flash to generate an answer to the corresponding MCQ and open-ended question using only that frame. We then use Gemini-2.0-Flash to judge the open-ended answers. Questions for which Gemini-2.0-Flash produces a correct response on both MCQ and open-ended formats are excluded from the benchmark. This filtering step ensures that the remaining questions require broader temporal understanding and cannot be resolved using minimal visual context."

    Section 4.3 reports as an empirical discovery that all models score only about 10% with one frame and that VIDEOEVAL-PRO cannot be solved without richer visual cues. But Section 3.1 constructed the benchmark by explicitly deleting every question that Gemini-2.0-Flash could answer correctly from a single frame. The property later presented as a finding, namely that the benchmark resists single-frame answering and therefore rewards additional frames, is built into the selection rule. The exact accuracy values and cross-model rankings remain empirical, so the circularity is partial rather than total.

full rationale

The most load-bearing circular step is the difficulty filter. The benchmark is defined to exclude single-frame-answerable and prior-answerable items, and the paper then cites low single-frame accuracy and monotonic frame scaling as evidence that VIDEOEVAL-PRO 'requires rich temporal information' and measures true long-video understanding. That specific finding is partially an artifact of the construction: a benchmark that removes easy, single-frame, and prior-driven questions will, by design, show low one-frame scores and benefit from more frames. The MCQ-versus-open-ended drop is not circular, because it is a direct comparison of two answer formats on the same questions, and the absolute open-ended scores of the 21 models are genuine external measurements. The authors' self-citations to Vamba, VISTA, Mantis, and Video-SimpleQA are not load-bearing: they provide evaluation criteria or related work, not the justification for the central claim. A separate correctness risk, though not itself a circularity, is that Figure 4 contains a retained question whose gold answer comes from world knowledge ('Thousands of Canadian flags') and whose filter-model response states that it cannot be answered from the provided video, showing that the prior-removal guarantee is not actually met. That failure weakens the benchmark's validity but is a filtering defect rather than a reductive derivation; the reductive step is the use of the filter as proof of temporal-requirement findings.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on several manual thresholds and model-based filters. The single-frame Gemini-2.0-Flash filter is particularly important because it is chosen by the authors rather than derived from first principles, and it directly shapes the paper's frame-scaling findings.

free parameters (4)
  • video duration threshold = 10 minutes
    Questions from videos shorter than 10 minutes are discarded (Section 3.1). This manual threshold shapes the benchmark's duration distribution and difficulty.
  • option word count threshold = 5 words
    Questions whose average option word count exceeds 5 are dropped (Section 3.1), a manual design choice intended to keep answers concise for LLM judging.
  • single-frame filter model = Gemini-2.0-Flash
    Difficulty filtering uses this model to identify and remove questions answerable from one random frame (Section 3.1). The choice of filter model affects which questions remain and the benchmark's observed frame-scaling behavior.
  • LLM judge model = GPT-4o-0806
    All open-ended answers are graded by this specific model version (Section 3.3). The judge's strictness determines reported accuracy and the size of the MCQ-to-open gap.
assumptions (4)
  • domain assumption Converting an MCQ to a free-form question preserves the ground truth of the original question and yields a uniquely answerable question.
    The entire pipeline relies on this, stated in Section 3.1 Data Collection. The Toronto memorial example in Section 4.4 shows it can fail.
  • domain assumption The original benchmarks' gold answers are correct and unambiguous when used as open-ended reference answers.
    The paper reuses reference answers from Video-MME, MLVU, LVBench, and LongVideoBench without re-validation (Section 3.1).
  • domain assumption LLM-as-a-judge (GPT-4o) accurately measures correctness of short answers under the SimpleQA rubric.
    Evaluation pipeline in Section 3.3. The appendix acknowledges judge biases and version inaccessibility.
  • ad hoc to paper A question answerable from a single random frame by Gemini-2.0-Flash is also answerable by the evaluated models, so removing it is sufficient to ensure the benchmark requires temporal understanding.
    This is the difficulty filter (Section 3.1); it is an ad hoc proxy for 'not answerable without video' and is a key source of the frame-scaling result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation." pith.science (2026). https://pith.science/paper/QFQWN44T

@misc{pith2026250514640,
  author       = {Pith},
  title        = {Pith review of: VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFQWN44T}},
  note         = {Machine review of arXiv:2505.14640}
}
abstract

Large multimodal models (LMMs) have recently emerged as a powerful tool for long video understanding (LVU), prompting the development of standardized LVU benchmarks to evaluate their performance. However, our investigation reveals a rather sober lesson for existing LVU benchmarks. First, most existing benchmarks rely heavily on multiple-choice questions (MCQs), whose evaluation results are inflated due to the possibility of guessing the correct answer; Second, a significant portion of questions in these benchmarks have strong priors to allow models to answer directly without even reading the input video. For example, Gemini-1.5-Pro can achieve over 50\% accuracy given a random frame from a long video on Video-MME. We also observe that increasing the number of frames does not necessarily lead to improvement on existing benchmarks, which is counterintuitive. As a result, the validity and robustness of current LVU benchmarks are undermined, impeding a faithful assessment of LMMs' long-video understanding capability. To tackle this problem, we propose VideoEval-Pro, a realistic LVU benchmark containing questions with open-ended short-answer, which truly require understanding the entire video. VideoEval-Pro assesses both segment-level and full-video understanding through perception and reasoning tasks. By evaluating 21 proprietary and open-source video LMMs, we conclude the following findings: (1) video LMMs show drastic performance ($>$25\%) drops on open-ended questions compared with MCQs; (2) surprisingly, higher MCQ scores do not lead to higher open-ended scores on VideoEval-Pro; (3) compared to other MCQ benchmarks, VideoEval-Pro benefits more from increasing the number of input frames. Our results show that VideoEval-Pro offers a more realistic and reliable measure of long video understanding, providing a clearer view of progress in this domain.

Figures

Figures reproduced from arXiv: 2505.14640 by the authors.

Figure 1
Figure 1. Comparison between VIDEOEVAL-PRO and MCQ benchmarks. Left: MCQ benchmarks yield inflated scores on identical questions (MCQ vs. Open) and can misrepresent model performance (LVBench [1]). Right: VIDEOEVAL-PRO cannot be effectively solved with a single input frame, and performance scales consistently with more frames. Video-MME [2] exhibits contradictory trends. extending their context length [6, 7], dropping or merg… view at source ↗
Figure 2
Figure 2. Summary of VIDEOEVAL-PRO data composition and task type distribution. to limited answerability. LongVideoBench, which is smaller in scale and subject to strict selection criteria, contributes 36 questions (3%). This diverse composition ensures that VIDEOEVAL-PRO spans a wide range of content domains and video types, ensuring a comprehensive model evaluation. Task Definition and Distribution Given the questions we co… view at source ↗
Figure 3
Figure 3. Comparison between VIDEOEVAL-PRO and Video-MME accuracy across five LMMs. in Figure 3a. For comparison, we also present the corresponding results from the Video-MME benchmark using the same models and frame settings in Figure 3b. Our first observation is that existing benchmarks such as Video-MME yield relatively high accuracy even when only one frame is provided to the model. As shown in Figure 3b, both proprietary… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons between VIDEOEVAL-PRO and the corresponding MCQ problems. selects the correct answer in the MCQ setting but fails to produce accurate factual details in the free-form response. The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TrajTok: Learning Trajectory Tokens enables better Video Understanding

    cs.CV 2026-02 unverdicted novelty 7.0 of 10

    TrajTok learns to tokenize video into object-trajectory tokens end-to-end, improving video CLIP, probing, and VLM performance over patch and token-merging baselines.

  2. Vid-SME: Membership Inference Attacks against Large Video Understanding Models

    cs.CV 2025-05 reject novelty 7.0 of 10

    Vid-SME computes Sharma-Mittal entropy differences between natural and reversed video frame sequences to infer training membership in video understanding LLMs, but its effectiveness is confounded by member/non-member ...

  3. VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.

  4. Video-Oasis: Rethinking Evaluation of Video Understanding

    cs.CV 2026-03 conditional novelty 6.0 of 10

    About 55% of existing video-understanding benchmark samples are solvable without vision or time; on the filtered remainder, SOTA models perform near random chance.

  5. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

  6. Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-XL-2 cuts long-video inference cost with chunked pre-filling and query-gated dense-or-sparse KV reloading, reporting half the FLOPs and a third less decoding memory at roughly equal benchmark scores.

Reference graph

Works this paper leans on

55 extracted references · 7 canonical work pages · cited by 6 Pith papers

  1. [1]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024

    arXiv 2024

  2. [2]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

  3. [3]

    Video anomaly detection and explanation via large language models

    Hui Lv and Qianru Sun. Video anomaly detection and explanation via large language models. arXiv preprint arXiv:2401.05702, 2024

    arXiv 2024

  4. [4]

    Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset

    Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710–9719, 2021

    2021

  5. [5]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  6. [6]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024

    arXiv 2024

  7. [7]

    Longvila: Scaling long-context visual language models for long videos.arXiv preprint arXiv:2408.10188, 2024

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos.arXiv preprint arXiv:2408.10188, 2024

    arXiv 2024

  8. [8]

    Videochat-flash: Hierarchical compression for long-context video modeling.arXiv preprint arXiv:2501.00574, 2024

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling.arXiv preprint arXiv:2501.00574, 2024

    arXiv 2024

Show all 55 references
  1. [9]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025

    2025 arXiv

  2. [10]

    Vamba: Under- standing hour-long videos with hybrid mamba-transformers.arXiv preprint arXiv:2503.11579, 2025

    Weiming Ren, Wentao Ma, Huan Yang, Cong Wei, Ge Zhang, and Wenhu Chen. Vamba: Under- standing hour-long videos with hybrid mamba-transformers.arXiv preprint arXiv:2503.11579, 2025

    2025 arXiv

  3. [11]

    Token-efficient long video understanding for multimodal llms.arXiv preprint arXiv:2503.04130, 2025

    Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, et al. Token-efficient long video understanding for multimodal llms.arXiv preprint arXiv:2503.04130, 2025

    2025

  4. [12]

    Bimba: Selective-scan compression for long-range video question answering.arXiv preprint arXiv:2503.09590, 2025

    Md Mohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Gedas Bertasius, and Lorenzo Torresani. Bimba: Selective-scan compression for long-range video question answering.arXiv preprint arXiv:2503.09590, 2025

    2025 arXiv

  5. [13]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  6. [14]

    Vista: Enhancing long- duration and high-resolution video understanding by video spatiotemporal augmentation.arXiv preprint arXiv:2412.00927, 2024

    Weiming Ren, Huan Yang, Jie Min, Cong Wei, and Wenhu Chen. Vista: Enhancing long- duration and high-resolution video understanding by video spatiotemporal augmentation.arXiv preprint arXiv:2412.00927, 2024

    2024 arXiv

  7. [15]

    Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  8. [16]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025. 10

    2025 arXiv

  9. [17]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  10. [18]

    Video-xl-pro: Re- constructive token compression for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025

    Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Re- constructive token compression for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025

    2025 arXiv

  11. [19]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  12. [20]

    Longvideobench: A benchmark for long- context interleaved video-language understanding.Advances in Neural Information Processing Systems, 37:28828–28857, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding.Advances in Neural Information Processing Systems, 37:28828–28857, 2024

    2024

  13. [21]

    Hourvideo: 1-hour video- language understanding.Advances in Neural Information Processing Systems, 37:53168–53197, 2024

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding.Advances in Neural Information Processing Systems, 37:53168–53197, 2024

    2024

  14. [22]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  15. [23]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  16. [24]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  17. [25]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  18. [26]

    Mantis: Interleaved multi-image instruction tuning.arXiv preprint arXiv:2405.01483, 2024

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning.arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  19. [27]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  20. [28]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  21. [29]

    Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    2024 arXiv

  22. [30]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  23. [31]

    Longvu: Spa- tiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spa- tiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    arXiv

  24. [32]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14485, 2024. 11

    2024 arXiv

  25. [33]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  26. [34]

    Vript: A video is worth thousands of words.Advances in Neural Information Processing Systems, 37:57240–57261, 2024

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words.Advances in Neural Information Processing Systems, 37:57240–57261, 2024

    2024

  27. [35]

    Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    2024 arXiv

  28. [36]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017

    2017

  29. [37]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. InProceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019

    2019

  30. [38]

    Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  31. [39]

    Videovista: A versatile benchmark for video understanding and reasoning.arXiv preprint arXiv:2406.11303, 2024

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning.arXiv preprint arXiv:2406.11303, 2024

    2024 arXiv

  32. [40]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024

    2024 arXiv

  33. [41]

    Video simpleqa: Towards factuality evaluation in large video language models.arXiv preprint arXiv:2503.18923, 2025

    Meng Cao, Pengfei Hu, Yingyao Wang, Jihao Gu, Haoran Tang, Haoze Zhao, Jiahua Dong, Wangbo Yu, Ge Zhang, Ian Reid, et al. Video simpleqa: Towards factuality evaluation in large video language models.arXiv preprint arXiv:2503.18923, 2025

    2025 arXiv

  34. [42]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  35. [43]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  36. [44]

    Gpt-4o.https://openai.com/index/hello-gpt-4o/, 2024

    OpenAI. Gpt-4o.https://openai.com/index/hello-gpt-4o/, 2024

    2024

  37. [45]

    Gpt-4o mini: Advancing cost-efficient intelligence, July 2024

    OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence, July 2024. URL https:// openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ . Ac- cessed: 2025-05-08

    2024

  38. [46]

    Introducing gpt-4.1 in the api, April 2025

    OpenAI. Introducing gpt-4.1 in the api, April 2025. URL https://openai.com/index/ gpt-4-1/. Accessed: 2025-05-08

    2025

  39. [47]

    Gemini 2.5: Our most intelligent ai model, March

    Koray Kavukcuoglu. Gemini 2.5: Our most intelligent ai model, March

  40. [48]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 12

    2024 arXiv

  41. [49]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  42. [50]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  43. [51]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    2024 arXiv

  44. [52]

    Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture.arXiv preprint arXiv:2409.02889, 2024

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture.arXiv preprint arXiv:2409.02889, 2024

    2024

  45. [53]

    Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023. 13 Appendix A Limitations Our VIDEOEVAL-PROemploys the LLM-as-a-Judge paradigm and therefore inherits certain limita- tions from this judging paradi...

    2023 arXiv

  46. [55]

    Keep" if the question can be answered by someone who has watched the video, even if the answer requires reasoning or summarizing visual or auditory evidence. -

    to accelerate the inference speed of the LMMs. As the main bottleneck for our evaluation comes from the decoding speed of very long videos, we pre-extract all the video frames from the source video and directly load the frame images during evaluation. A 7B-scale model takes ap...

    1998

  47. [2025]

    Accessed: 2025-05-08

    URL https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/. Accessed: 2025-05-08

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.