Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark claims the human–AI gap in video understanding is real, not a frame-sampling artefact.

desk verdict A genuinely useful benchmark with a plausible but provisional headline gap; the 80-frame sampling check and an LLM judge need scrutiny before the numbers are taken at face value. read the letter →

arxiv 2507.15028 v1 pith:5KY3RBFX submitted 2025-07-20 cs.CV

Yuanhan Zhang , Yunice Chew , Yuhao Dong , Aria Leo , Bo Hu , Ziwei Liu This is my paper
classification cs.CV
keywords videoreasoningbenchmarklargelanguagemodelsnaturaladversarialrobustnessvisualcomplexitynarrativeframesamplingopen-endedquestionansweringhuman-modelperformancegap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the Video Thinking Test (Video-TT), a benchmark of 1,000 short YouTube videos built to ask whether video large language models genuinely understand real-world clips, with each video carrying one primary open-ended question and four 'natural adversarial' variants that rephrase, cue, or mislead the same query. The authors' aim is to measure two capacities separately: correctness on complex visual narratives and robustness under realistic wording changes, and to do so in a way that rules out frame sampling as the cause of model failures. The headline result is a wide gap: humans reach 84.3% accuracy and 64.3% robustness, while the best model tested, GPT-4o, reaches 36.6% accuracy and 36.0% robustness on the same items. If the benchmark's controls hold, this means current video AI is roughly half as accurate as people on complex short clips and far more easily thrown off by how a question is phrased.

What carries the argument

The mechanism that carries the argument is the curation pipeline, anchored by the Sampling Check: each retained question must be answerable from 80 uniformly sampled frames, which converts the benchmark from a test of frame selection into a test of understanding. Question design rests on eight complexity factors — four visual (unclear or unusual content, movement speed, spatial-temporal arrangement, illusions) and four narrative (complex plot, narrative editing, technical editing, world knowledge) — organized around a three-level content hierarchy of element, event, and plot. Each primary question is expanded into four adversarial rewrites, and robustness is scored as the ratio $R = |A_{\text{full correct}}| / |A_{\text{primary correct}}|$, the share of videos where a model that got the primary question right also got all four variants right. A triple-annotator alignment check and a complexity filter that requires at least one of three probe models to fail the question are intended to keep the retained items genuinely hard.

What would settle it

Take a random sample of Video-TT questions, give the same models the complete videos instead of 80 frames, and have an independent human panel redo the Sampling Check on the retained questions. If model accuracy rises sharply with full frames, or if the panel judges a meaningful share of retained questions under-specified from the 80-frame set, the paper's core claim that errors are comprehension failures rather than sampling failures would be falsified.

Watch

Extended reading notes

Core claim

The paper claims that the gap between video LLMs and human video understanding is real and mostly a matter of comprehension, not of which frames a model happens to look at. To make that claim, every question in Video-TT must pass a Sampling Check: annotators confirm the question is answerable from 80 uniformly sampled frames with audio removed, so a wrong answer cannot be blamed on missing information. Under that design the top proprietary model, GPT-4o, answers 36.6% of primary open-ended questions correctly against 84.3% for humans, and its robustness score — the fraction of videos where a correct primary answer is followed by four correct adversarial variants — is 36.0% against 64.3% for humans. Open-source models match GPT-4o on multiple-choice accuracy but trail by up to 13.8 points on the adversarial track. GPT-4o's error analysis attributes most failures to spatio-temporal confusion (79% of localization errors), missing world knowledge (44% of character-reaction errors), and an inability to bind separate scenes into one causal story (55% of plot-causality errors).

Load-bearing premise

The load-bearing premise is the Sampling Check: that an annotator's judgment that a question is answerable from 80 uniformly spaced frames is reliable, so every error the benchmark counts is a genuine understanding failure rather than a failure of the frames to contain what the question asks about.

Editorial extensions

If this is right

  • If the gap is genuine, current video LLMs are about half as accurate as humans on complex short-form video reasoning, and simply feeding more frames will not close it: model accuracy saturates near 8 frames while human accuracy keeps climbing to near-perfect at 64.
  • Open-ended questions expose failures that multiple-choice tests hide: the best open-source model matches GPT-4o on multiple-choice (47.5 vs 46.6) but trails badly on primary open-ended questions (24.4 vs 36.6).
  • Natural adversarial robustness is a distinct capability axis: GPT-4o holds a 36.0% robustness score against 64.3% for humans, and the strongest open-source model trails it by 13.8 points.
  • Two cheap interventions help only partially: chain-of-thought prompting improves resistance to misleading wording by about 6.8% relative, and adding audio transcripts lifts robustness by roughly 15% relative, while neither moves multiple-choice accuracy.
  • Question types that stress counting and displacement show a negative correlation between model and human performance (r = -0.50), so the benchmark's hardest items for machines are among the easiest for people.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference, not the paper's: a direct stress test would re-run the same models on Video-TT with full videos instead of 80 frames; flat accuracy would confirm the paper's sampling-check premise, while a sharp rise would show the gap is partly a frame-rationing artefact.
  • The paper does not condition the robustness score on each variant separately, so one could extend it to ask whether misleading rephrasings and helpful rephrasings are independent failure modes for a given model.
  • Because open-ended answers are graded by another language model, part of the measured gap may be the grader's own reasoning limits; a human-graded subsample would reveal whether the LLM judge under-credits unusual but correct answers.
  • The negative human–model correlation across question types suggests a concrete trajectory for progress: models should become more human-like by strengthening counting and displacement tracking rather than illusion handling, a prediction that per-type accuracy curves could track over successive model generations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Video-TT, a benchmark of 1,000 YouTube Shorts videos, each with one primary open-ended question and four natural adversarial questions (5,000 QA pairs total). The curation pipeline is designed to retain questions that are complex for humans and current models and that, according to an annotator check, are answerable from 80 uniformly sampled frames without audio. The main empirical claim, stated in Section 1, is that humans score 84.3% accuracy and 64.3% robustness, while the best evaluated model, GPT-4o, scores 36.6% and 36.0%; open-source models are comparable on accuracy but lag on robustness. The paper also contributes an error analysis of GPT-4o failures across question types.

Significance. If the sampling check and the human baseline are valid, Video-TT is a useful benchmark because it directly addresses a known confound in video benchmarks—frame sampling—and it provides a multi-question robustness protocol with a human reference. The dataset includes rationales, a transparent curation pipeline, and a falsifiable head-to-head comparison. The main numerical result, if correct, substantiates a large remaining gap between video LLMs and humans on short real-world videos. However, the benchmark's central separation of sampling errors from comprehension errors rests on two unverified assumptions (the 80-frame sufficiency check and the comparability of human and model viewing conditions), and the open-ended scoring relies on a single LLM judge without reported human-agreement measurements. These issues do not destroy the contribution, but they must be addressed before the headline numbers can be taken at face value.

major comments (4)
  1. [Sec. 3.3, Sampling Check] The claim that every retained question is answerable from 80 uniformly sampled frames is load-bearing for the paper's central message that model errors are comprehension errors rather than sampling errors (Sec. 1 and Fig. 2, right). The check as described is an annotator instruction plus a self-report: only 44 of 1,344 items were rejected at that stage (Supp. Fig. 10), and no independent verification is reported. This matters because the dataset deliberately includes event counting, displacement, and spatial-temporal arrangement (Fig. 4, Sec. 3.1), where the needed evidence is often transient or non-uniform. I request an independent verification protocol (for example, a second set of annotators answering each question from exactly the 80 frames while blinded to the full video, with agreement and success rates reported), or an analysis demonstrating that the correct answers of the retained questions are visible in the 80 frames.
  2. [Sec. 3.3 / 4.1, Human baseline] The human-vs-model comparison assumes identical stimuli. The paper never states whether human evaluators saw the full video with audio or the same 80-frame, no-audio condition used for models. Fig. 9(c) shows that human accuracy depends strongly on frame count, rising to near-perfect accuracy at 64 frames, so this is not a minor detail. If the human baseline used the full video, the reported accuracy and robustness gaps are partly explained by information access, and the 80-frame sampling check becomes the only bridge supporting the comparison. Please specify the exact human viewing condition and, ideally, add a human 80-frame no-audio condition.
  3. [Sec. 4.1, Metric] Open-ended answers are scored by Qwen2.5-72B with a fixed prompt and a threshold of 3 out of 5, but no human-model agreement is reported. Since the same broad model family (Qwen2.5-VL) also appears in the probe set used to filter questions for complexity (Sec. 3.3, "Ensuring Complexity for Model"), there is a coupling between dataset construction and evaluation. Please report judge agreement against human labels on a sample and a robustness check with an alternative judge or human scoring to show that the accuracy numbers are not judge-specific.
  4. [Table 1] The headline comparison is presented without uncertainty quantification. With 1,000 primary questions, differences of a few percentage points between models (for example, LLaVA-Video-72B at 37.6% versus GPT-4o at 45.2% average accuracy, or the open-source versus GPT-4o robustness gap of 13.8 points) may or may not be statistically significant, and the robustness score has a smaller denominator because it only includes videos where the primary question is answered correctly. Report confidence intervals, bootstrap estimates, or significance tests for the main claims, and clarify how many videos contribute to each robustness score.
minor comments (4)
  1. [Throughout] The manuscript contains many typos and spacing errors, including "fild" and "langauge" in Section 2, "in addtion" in Section 4.2, "TemperalBench" in the dataset comparison table, and inconsistent uses of "Video LMM" versus "video LLM"; a careful copyedit is needed.
  2. [Fig. 3] The pipeline diagram contains unexplained fragments such as "&&" and "||", and the flow from one stage to the next is hard to follow; please make the diagram self-contained with clear labels and definitions.
  3. [Sec. 5] The text first says that five question categories are directly linked to specific complex factors and then says the error analysis is performed across the other 13 question types; please clarify whether the five categories are included or excluded from the reported percentages in Fig. 8.
  4. [Sec. 4.3 / Fig. 9(d)] The reported "15% relative gain" in robustness from audio transcripts should state which baseline it is relative to and which model(s) it applies to, since the main robustness comparison uses GPT-4o.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the headline human-vs-model gap is an empirical benchmark measurement; the model-filtered curation creates mild selection coupling but does not force the reported numbers.

full rationale

The paper's central claims are measurements on a constructed benchmark, not predictions derived from fitted parameters or self-citations. Section 3.3's 'Ensuring Complexity for Model' step filters questions using failures of GPT-4o, LLaVA-Video-7B, and Qwen2.5-VL-7B, and those same model families appear in Table 1; this is a real selection coupling that makes the benchmark intentionally hard for those models. However, the filter only requires that at least one of the three models fails at least once in three attempts, and the headline accuracies (humans 84.3%/64.3% vs GPT-4o 36.6%/36.0%) are contingent on answer scoring, model versions, and the 1,000 retained videos, so they are not forced by construction. The human baseline is an independent measurement on the same questions, and the robustness score is a definition, not a derived prediction. The 80-frame sampling check is a subjective curation criterion and a validity risk (questions involving counting, displacement, or spatial-temporal arrangement may need more than 80 frames), but it is an assumption about data quality, not a circular step. No load-bearing argument reduces to a self-citation; citations to the authors' prior models refer to publicly released checkpoints evaluated zero-shot. Therefore the derivation chain is self-contained and the paper is not circular, though the selection-filter coupling and sampling-check subjectivity are worth flagging as validity concerns.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The benchmark's central claims rest mainly on design choices rather than derived parameters. The most consequential are the 80-frame answerability criterion, the model-failure retention rule, and the use of a language-model judge for open-ended scoring. These choices are reasonable but are not independently validated, and the first two inflate the reported human-model gap.

free parameters (2)
  • Uniform frame budget = 80 frames
    Hand-chosen number of uniformly sampled frames used as the answerability check; central to the paper's claim that errors are not caused by insufficient sampling, but no ablation justifies it.
  • Correctness threshold = 3 out of 5
    Open-ended answers are judged correct when Qwen2.5-72B assigns a score of 3 or higher; this threshold is not validated against human grading.
assumptions (6)
  • ad hoc to paper 80 uniformly sampled frames are sufficient for every retained question
    Sec. 3.3 Sampling Check instructs annotators to formulate questions answerable from 80 frames; the claim that errors reflect understanding rather than sampling depends on this unverified premise.
  • ad hoc to paper A question is 'complex' only if at least one of three probe models fails it in at least one of three attempts
    Sec. 3.3 Ensuring Complexity for Model; this retention rule makes the benchmark adversarial to existing models and is not a property of video difficulty in general.
  • domain assumption Qwen2.5-72B text-only judge can correctly decide whether open-ended answers match the gold answer
    Appendix 3 defines the grading prompt; no human agreement study is provided, so grade validity is assumed.
  • domain assumption The eight complexity factors are sufficient to characterize what makes video questions hard
    Sec. 3.1 derives visual and narrative complexity from cognitive science references; the list is a modeling choice, not a proven taxonomy.
  • domain assumption Annotator consensus is reliable ground truth
    Sec. 3.3 Alignment Check requires unanimous agreement for causal questions; human judgment is treated as ground truth without inter-annotator reliability statistics.
  • domain assumption The filtered YouTube Shorts sample is representative enough for general conclusions
    Sec. 3.4 filters violent, explicit, and suspected AI-generated videos, all under 65 seconds; sample skew (comedy, sports, daily life) is not analyzed for representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding." pith.science (2026). https://pith.science/paper/5KY3RBFX

@misc{pith2026250715028,
  author       = {Pith},
  title        = {Pith review of: Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5KY3RBFX}},
  note         = {Machine review of arXiv:2507.15028}
}
read the original abstract

Human intelligence requires correctness and robustness, with the former being foundational for the latter. In video understanding, correctness ensures the accurate interpretation of visual content, and robustness maintains consistent performance in challenging conditions. Despite advances in video large language models (video LLMs), existing benchmarks inadequately reflect the gap between these models and human intelligence in maintaining correctness and robustness in video interpretation. We introduce the Video Thinking Test (Video-TT), to assess if video LLMs can interpret real-world videos as effectively as humans. Video-TT reflects genuine gaps in understanding complex visual narratives, and evaluates robustness against natural adversarial questions. Video-TT comprises 1,000 YouTube Shorts videos, each with one open-ended question and four adversarial questions that probe visual and narrative complexity. Our evaluation shows a significant gap between video LLMs and human performance.

Figures

Figures reproduced from arXiv: 2507.15028 by the authors.

Figure 1
Figure 1. Overview of the Video Thinking Test. Video-TT introduces two challenges: (1) ensuring correctness in understanding complex visual stories; (2) maintaining robustness against natural adversarial conditions. Abstract Human intelligence requires correctness and robustness, with the former being foundational for the latter. In video understanding, correctness ensures the accurate interpre￾tation of visual content, and r… view at source ↗
Figure 3
Figure 3. Benchmark Curation Pipeline. Our annotation pipeline ensures that each question: (1) is complex enough to differentiate between human and model video understanding capabilities; (2) can be understood with a limited number of sampled frames; (3) also assesses the models’ robustness against natural adversarial conditions. Q-5: Why did the woman bump into the door instead of opening it herself at the end of the video? … view at source ↗
Figure 5
Figure 5. VQA Question Prototypes. We present our five ques￾tion prototypes. Man highlights the man framed by a bounding box. 3.2. How to Ask a Natural Adversarial Question? To reach human-level understanding of videos, it is not enough to just answer questions correctly; we must also ex￾plore how changing the wording of a question affects model performance. These natural adversarial questions broaden our study and help users… view at source ↗
Figures from the paper (6 more)
Figure 6
Figure 6. Figure 6: Benchmark Statistics. Left: We show our 18 question by category alongside the number of questions. Right: The his￾togram of the video length. 3.4. Dataset Statistics Overall, we collected 5,000 questions and answers for 1,000 videos. Initially, questions were classifie…
Figure 7
Figure 7. Figure 7: Error cases in typical question types. We mark rationale answers with a grey background. Video links of each case : Q-1, Q-2, Q-3, Q-4. 0 20 40 60 80 100 Percentage Plot Attributes+Objective Causality Character Reaction/Motivation Element/Event Localization Positional …
Figure 8
Figure 8. Figure 8: Human-conducted analysis of errors by question type. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: (a-b) Comparison of human and average model performance based on correctness and robustness across question types. (c) [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: The data annoation flow of Video Turing Test. Q stands for Question; A stands for Answer; R stands for Rationale. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video Reasoning without Training

    cs.CV 2025-10 conditional novelty 6.0 of 10

    An entropy-guided, inference-time value-cache controller improves video reasoning accuracy and cuts output tokens versus RL-trained baselines.

Reference graph

Works this paper leans on

58 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 3

    work page 2015

  2. [2]

    Temporalbench: Towards fine-grained temporal understanding for multimodal video models

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. Temporalbench: Towards fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818, 2024. 1, 2, 3

    arXiv

  3. [3]

    Collecting highly paral- lel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly paral- lel data for paraphrase evaluation. InProceedings of the 49th annual meeting of the association for computational linguis- tics: human language technologies , pages 190–200, 2011. 3

    work page 2011

  4. [4]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 1, 5

    arXiv 2024

  5. [5]

    Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.14432, 2024. 8

    arXiv 2024

  6. [6]

    Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing. arXiv preprint arXiv:2406.14515, 2024. 1, 3, 8

    arXiv 2024

  7. [7]

    Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 1, 2, 3, 5, 8

    arXiv 2024

  8. [8]

    Agqa: A benchmark for compositional spatio-temporal reasoning

    Madeleine Grunde-McLaughlin, Ranjay Krishna, and Ma- neesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11287–11297, 2021. 3

    work page 2021

Show all 58 references
  1. [9]

    Similarity and fea- tures of natural textures

    Christopher Heaps and Stephen Handel. Similarity and fea- tures of natural textures. Journal of Experimental Psychol- ogy: Human Perception and Performance, 25(2):299, 1999. 3

    1999

  2. [10]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 15262–15271, 2021. 2

    2021

  3. [11]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline pro- fessional videos, 2025

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline pro- fessional videos, 2025. 1, 2, 3

    2025

  4. [12]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2758–2766, 2017. 3

    2017

  5. [13]

    Robust modeling in cognitive science

    Michael D Lee, Amy H Criss, Berna Devezer, Christopher Donkin, Alexander Etz, F ´abio P Leite, Dora Matzke, Jef- frey N Rouder, Jennifer S Trueblood, Corey N White, et al. Robust modeling in cognitive science. Computational Brain & Behavior, 2:141–153, 2019. 1

    2019

  6. [14]

    Tvqa: Localized, compositional video question answering

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696, 2018. 3

    2018 arXiv

  7. [15]

    Mvbench: A comprehensive multi- modal video understanding benchmark, 2023

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark, 2023. 1, 2, 3

    2023

  8. [16]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  9. [17]

    Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024. 2, 3

    2024 arXiv

  10. [18]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Ji- wen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024. 1, 5

    2024 arXiv

  11. [19]

    Chain-of-spot: Interactive reasoning improves large vision-language models

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Ji- wen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403.12966,

    arXiv

  12. [20]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04328, 2025. 1, 5

    2025 arXiv

  13. [21]

    Video detail caption, 2024

    LMMs-Lab. Video detail caption, 2024. 8

    2024

  14. [22]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. 1

    2024

  15. [23]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36, 2024. 2

    2024

  16. [24]

    Identifying the perceptual dimensions of visual complexity of scenes

    Aude Olivia, Michael L Mack, Mochan Shrestha, and An- gela Peeper. Identifying the perceptual dimensions of visual complexity of scenes. In Proceedings of the annual meeting of the cognitive science society, 2004. 3

    2004

  17. [25]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https : / / openai . com / index/hello-gpt-4o/, 2024. 5

    2024

  18. [26]

    Robustness analysis of video- language models against visual and language perturbations

    Madeline Schiappa, Shruti Vyas, Hamid Palangi, Yogesh Rawat, and Vibhav Vineet. Robustness analysis of video- language models against visual and language perturbations. Advances in Neural Information Processing Systems , 35: 34405–34420, 2022. 2, 3

    2022

  19. [27]

    Visual cot: Advancing multi-modal language models with a com- prehensive dataset and benchmark for chain-of-thought rea- soning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuo- fan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a com- prehensive dataset and benchmark for chain-of-thought rea- soning. Advances in Neural Information Processing Sys...

    2024

  20. [28]

    Complex narratives

    Jan Simons. Complex narratives. In Hollywood puzzle films, pages 17–34. Routledge, 2014. 4

    2014

  21. [29]

    A standardized set of 260 pictures: norms for name agreement, image agree- ment, familiarity, and visual complexity

    Joan G Snodgrass and Mary Vanderwart. A standardized set of 260 pictures: norms for name agreement, image agree- ment, familiarity, and visual complexity. Journal of experi- mental psychology: Human learning and memory, 6(2):174,

  22. [30]

    Visual agents as fast and slow thinkers

    Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers. arXiv preprint arXiv:2408.08862, 2024. 8

    2024 arXiv

  23. [31]

    Curious objects: How vi- sual complexity guides attention and engagement

    Zekun Sun and Chaz Firestone. Curious objects: How vi- sual complexity guides attention and engagement. Cognitive Science, 45(4):e12933, 2021. 3

    2021

  24. [32]

    Cognitive load during problem solving: Ef- fects on learning

    John Sweller. Cognitive load during problem solving: Ef- fects on learning. Cognitive science, 12(2):257–285, 1988. 4

    1988

  25. [33]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 5

    2023 arXiv

  26. [34]

    Qwen2.5-vl, 2025

    Qwen Team. Qwen2.5-vl, 2025. 1, 5

    2025

  27. [35]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. 8

    2023

  28. [36]

    Star: A benchmark for situated reasoning in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 3

    2021

  29. [37]

    Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024. 1, 3, 5

    2024

  30. [38]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 2, 3

    2021

  31. [39]

    Funqa: Towards surprising video comprehension

    Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. arXiv preprint arXiv:2306.14899, 2023. 3

    2023 arXiv

  32. [40]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In Proceedings of the 25th ACM international confer- ence on Multimedia, pages 1645–1653, 2017

    2017

  33. [41]

    Sutd-trafficqa: A question answering benchmark and an efficient network for video rea- soning over traffic events

    Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient network for video rea- soning over traffic events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9878–9888, 2021

    2021

  34. [42]

    Clevrer: Collision events for video representation and reasoning

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019

    1910 arXiv

  35. [43]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019

    2019

  36. [44]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127–9134, 2019. 3

    2019

  37. [45]

    Social-iq: A question answer- ing benchmark for artificial social intelligence

    Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answer- ing benchmark for artificial social intelligence. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807–8817, 2019. 2

    2019

  38. [46]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 1

    2023 arXiv

  39. [47]

    B- avibench: Towards evaluating the robustness of large vision- language model on black-box adversarial visual-instructions,

    Hao Zhang, Wenqi Shao, Hong Liu, Yongqiang Ma, Ping Luo, Yu Qiao, Nanning Zheng, and Kaipeng Zhang. B- avibench: Towards evaluating the robustness of large vision- language model on black-box adversarial visual-instructions,

  40. [48]

    Lmms- eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms- eval: Reality check on the evaluation of large multimodal models, 2024. 5

    2024

  41. [49]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 8

    2024 arXiv

  42. [50]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 1

    2024

  43. [51]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 1, 5

    2024

  44. [52]

    Worldqa: Multimodal world knowledge in videos through long-chain reasoning, 2024

    Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christo- pher Arif Setiadharma, Jingkang Yang, and Ziwei Liu. Worldqa: Multimodal world knowledge in videos through long-chain reasoning, 2024. 1, 2, 3

    2024

  45. [53]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  46. [54]

    Hierarchical video content description and summarization using unified semantic and visual similarity

    Xingquan Zhu, Jianping Fan, Ahmed K Elmagarmid, and Xindong Wu. Hierarchical video content description and summarization using unified semantic and visual similarity. Multimedia Systems, 9:31–53, 2003. 3 Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reas...

    2003

  47. [55]

    In total, the annotation process cost 8227.32 human hours

    Annotation Detail We present the number of human hours at each stage in the Data Curation Process as follows. In total, the annotation process cost 8227.32 human hours. Table 2. Time Estimation for Dataset Curation Process. Notes: *2 indicates that two people are required for ...

  48. [56]

    • Aparaphrased correct be the set of videos where the para- phrased open-ended question is answered correctly

    Mathematical Definition of the Robustness Score • Aprimary correct be the set of videos where the primary open- ended question is answered correctly. • Aparaphrased correct be the set of videos where the para- phrased open-ended question is answered correctly. • Acorrectly led...

  49. [57]

    3 shows the prompt for evaluating open-ended an- swers

    Prompt for Evaluating Open-ended Answer Table. 3 shows the prompt for evaluating open-ended an- swers. A score of 3 or higher is considered correct, while scores below 3 are deemed incorrect. We refer to the prompt introduced in VideoChatGPT [22]. System Message You are an int...

  50. [58]

    element” and “event

    Error Analysis In this section, we give more analysis about the errors made by GPT-4o. Complex Video Collection [2977]Complex Video Collection [2977]Complex Video Collection [2977]Complex Video Collection [2977]Complex Video Collection [2977] Rejected Videos [639]Rejected Vide...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.