Pith. sign in

REVIEW 3 major objections 5 minor 81 references

Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MF^2, a benchmark of 53 full-length open-licensed movies with 868 manually written true/false claim pairs, shows that the best current vision-language models (Gemini 2.5 Pro at 60.6% pairwise accuracy) fall far short of human memory-based…

desk verdict MF2 is a genuinely useful benchmark for full-length movie narrative understanding, and the human-model gap is probably real, but the exact 24-point gap needs a cleaner human baseline before taking it at face value. read the letter →

arxiv 2506.06275 v1 pith:RFJBCMUM submitted 2025-06-06 cs.CV cs.CLcs.LG

Emmanouil Zaranis , António Farinhas , Saul Santos , Beatriz Canaverde , Miguel Moura Ramos , Aditya K Surikuchi , André Viveiros , Baohao Liao
show 23 more authors
Elena Bueno-Benito Nithin Sivakumaran Pavlo Vasylenko Shoubin Yu Sonal Sannigrahi Wafaa Mohammed Ben Peters Danae Sánchez Villegas Elias Stengel-Eskin Giuseppe Attanasio Jaehong Yoon Stella Frank Alessandro Suglia Chrysoula Zerva Desmond Elliott Mariella Dimiccoli Mohit Bansal Oswald Lanz Raffaella Bernardi Raquel Fernández Sandro Pezzelle Vlad Niculae André F. T. Martins
This is my paper
classification cs.CVcs.CLcs.LG
keywords long-formvideounderstandingvision-languagemodelsnarrativecomprehensionbenchmarkmoviecontrastiveclaimpairshumanevaluationfactandfib
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces $MF^{2}$, a benchmark for testing whether vision-language models truly understand full-length movies rather than retrieve isolated details. The dataset consists of 53 open-licensed movies, roughly 50-170 minutes long, matched with 868 hand-written pairs of claims: each pair contains one true statement (fact) and one minimally altered false statement (fib) about memorable narrative elements such as character motivation, emotion, causality, and event order. The evaluation protocol requires a model to mark both claims in a pair correctly, removing answer-ordering biases and forcing genuine discrimination. On this benchmark the best system, Gemini 2.5 Pro, reaches 60.6% pairwise accuracy, a 24.1 percentage-point gap behind a human baseline of 84.1%, while smaller open-weight models hover near chance. The paper's central claim is that current vision-language models lack the human ability to consolidate, retain, and reason over key narrative information over long time spans.

What carries the argument

The central object is the contrastive claim pair: a fact and a minimally edited fib that differ only in the words needed to flip the truth value, each targeting a single narrative element. Scoring is pairwise, so a response counts only if the model labels both claims in the pair correctly; this reduces reliance on superficial cues such as sentence length, order, or lexical similarity. Each pair is also labeled by reasoning granularity (single-scene, multi-scene, or global) and by comprehension dimension (event/entity understanding, temporal perception, emotion understanding, and causal reasoning), which lets the benchmark locate precisely where humans and models diverge.

What would settle it

Give a random sample of the 868 claim pairs to independent viewers who watch each film once and judge the claims from memory; if their pairwise accuracy falls well below 84.1%, the human ceiling that defines the gap is inflated. A model scoring above 84.1% pairwise on the same protocol would falsify the paper's conclusion that current vision-language models lack this ability.

Watch

Extended reading notes

Core claim

The paper claims that $MF^{2}$ measures genuine narrative comprehension of full-length movies and that current vision-language models fail it. On the benchmark's strict pairwise metric, where credit is given only when both the true and false claim in a pair are classified correctly, the strongest evaluated model reaches 60.6%, well below the 84.1% human baseline; open-weight models perform worse, with several large models near 50%. The paper also reports that models rely heavily on subtitles and on pretrained world knowledge: Gemini 2.5 Pro drops to 37.2% pairwise on video-only input, yet scores 43.7% with only the movie title and release year, suggesting that part of the measured performance is not visual narrative understanding at all.

Load-bearing premise

The central claim depends on the fact/fib labels being correct and on the co-author human evaluators, answering from memory without rewatching, being an unbiased upper bound; the paper itself notes that those evaluators may have remembered seeing the paired claim.

Editorial extensions

If this is right

  • The best closed model, Gemini 2.5 Pro, would need to gain roughly 24 percentage points in pairwise accuracy to match the human baseline, so the reported gap defines the current ceiling for machine narrative recall.
  • Adding subtitles changes results more than model scale: GPT-4o jumps from 18.8% to 46.8% pairwise with subtitles, showing that benchmark performance is heavily conditioned on textual grounding.
  • A model given only the movie title and release year scores above chance (43.7% pairwise), meaning a portion of the measured 'movie understanding' reflects pretrained world knowledge rather than comprehension of the video itself.
  • Human performance declines as reasoning moves from single-scene to multi-scene to global claims, while some models improve on global claims; this suggests humans and current models are not yet solving the same underlying task.
  • Emotion understanding is the clearest human advantage, reaching 90% pairwise for humans versus 52% for the best model, pointing to an underdeveloped capability in current vision-language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 868 surviving labels were filtered by author comments but not independently re-verified against the films, the reported 84.1% human upper bound could shift under a separate annotation study; a re-check of a random sample would show whether the human-model gap is overstated or understated.
  • The large gains from subtitles and the above-chance title-only score suggest that a stricter evaluation, hiding the movie title and using only visual frames, would better isolate visual narrative comprehension and likely compress current model scores.
  • Adopting the paper's own suggestion, future human evaluations could split each fact/fib pair across different annotators so that no single person sees both claims; the authors flag memory contamination as a limitation of their current protocol.
  • MF^2's design could be extended to episodic narrative video or to a memory-curve protocol in which models must identify when in the film the evidence for a claim appears, connecting the benchmark to research on memory consolidation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MF2, a benchmark for evaluating narrative comprehension of full-length movies. It consists of 53 public-domain movies (average 88.33 minutes) with 868 manually constructed contrastive claim pairs, each containing one true statement (fact) and one minimally edited false counterpart (fib). The authors evaluate several open-weight and closed vision-language models using a binary claim protocol, reporting pairwise accuracy (both claims in a pair must be correct) and standard accuracy. The main empirical claim is that the best model (Gemini 2.5 Pro with video and subtitles, 60.6% pairwise) falls far short of human performance (84.1% pairwise), indicating that current VLMs lack human-level ability to consolidate, reason over, and recall key narrative information from full-length films.

Significance. If the human baseline is reliable, MF2 addresses a genuine gap in long-form video benchmarks: it uses full-length movies, manually authored claims targeting narrative elements (motivations, causal chains, event order), a contrastive design that reduces surface cues, open-licensed data, and a public release of data and code. The benchmark also includes a useful ablation showing that subtitles substantially help models and that models rely on parametric knowledge. The manual annotation process is detailed in appendices, and the contrastive pair format is a thoughtful alternative to multiple-choice questions. However, the central contribution depends heavily on the validity of the reported human baseline, which is currently not established.

major comments (3)
  1. [§2.3, §3, Table 3] The reported 24.1-point gap between humans (84.1%) and the best model (60.6%) is not a symmetric comparison because human evaluators see both claims of each contrastive pair, whereas models are queried independently per claim. As the paper acknowledges in §7, annotators 'may have recalled previously seen claims from the same pair, potentially influencing their judgment on the latter.' Since fact and fib differ by minimal edits, a human who remembers the first claim can answer the second by consistency alone (e.g., if the first was True, the second must be False), effectively raising pairwise accuracy without additional movie comprehension. Models do not receive such pairing information. This asymmetry directly affects the headline gap and should be quantified. The authors should run a split-pair human evaluation (each annotator sees only one claim per pair) or report human accuracy on first-seen claims within each pair.
  2. [§2.3] The human baseline was collected from the research team itself: §2.2 states all 26 annotators are co-authors, and §2.3/acknowledgments show the human evaluators are also co-authors or acknowledged participants. No inter-annotator agreement is reported, no confidence intervals are given, and no significance tests accompany the model-vs-human comparison. Expert annotators who wrote or reviewed the claims are likely more motivated and more familiar with the task than naive viewers, potentially inflating the human score. The manuscript should report the number of annotators per movie, agreement statistics (e.g., Cohen's kappa), and ideally a separate evaluation with naive participants to establish an unbiased upper bound.
  3. [§2.3] The quality-control step filters 104 pairs based on annotator comments, but the remaining 868 gold labels are not independently verified. If some labels are incorrect or ambiguous, both human and model scores are affected, and the benchmark's validity as ground truth is undermined. The authors should provide a sample-based second annotation or an independent verification pass with agreement metrics, rather than relying solely on comments from the same annotators who produced the claims.
minor comments (5)
  1. [§2.2] Typographical error: 'constrastive' should be 'contrastive'.
  2. [Table 1] The 'Evaluation Format' column lists 'Claim pairs' for MF2; consider clarifying that this is a binary True/False claim judgment, distinct from the 'MC' and 'OE' formats listed for other datasets.
  3. [§3] Minor formatting issues: 'LLaV A-Video' appears with an inconsistent space in the model name, and 'VideoLLaMA3' is sometimes written as 'VideoLLaMA'.
  4. [§4.1] The sentence 'Smaller models perform only marginally above chance, with the best among them exceeding the random baseline by just 11.09%' does not identify which model achieves this value; please name the model or table row.
  5. [§1, §4.2] The paper contrasts its binary claim format with multiple-choice, but a two-alternative forced choice is still a form of selected-response format. The authors may want to explicitly justify why the contrastive structure, not the number of options, is the key difference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MF2 is an external benchmark evaluation with no fitted derivation chain or load-bearing self-citations.

full rationale

MF2 is a dataset and evaluation benchmark, not a model with a fitted derivation chain. The claims are human-authored, and every model score is produced by independent inference on the released benchmark with evaluation prompts and parsing rules; pairwise accuracy is computed post-hoc from per-claim predictions and is not used to set any model parameter. The human baseline is an external measurement, although it uses same-team evaluators, and the Section 7 acknowledgment that annotators may recall the counterpart claim is a stated limitation on comparability, not a circularity. The contrastive-pair protocol is attributed to external prior work [17], not to the present authors, and the only self-citations (e.g., [45] in related work) are descriptive references that do not carry the paper's central claim. No equation, fitted parameter, or derived result reduces by construction to the benchmark inputs, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on assumptions about label correctness, the validity of a single human baseline from the research team, and the representativeness of a public-domain film corpus. These are domain assumptions, not fitted parameters.

assumptions (4)
  • domain assumption Human memory recall of a movie after a single viewing is a valid operationalization of narrative understanding.
    The benchmark defines understanding via memorable claims that humans can recall without rewatching (Section 1, Fig. 1).
  • domain assumption The fact/fib labels written by the annotating co-authors are ground truth.
    Annotation guidelines instruct annotators to produce unambiguous claims (Section B.1); quality control removes 104 pairs but does not independently verify the remainder (Section 2.3).
  • domain assumption The corpus of 53 public-domain films from 1920-1970 is a representative testbed for long-movie understanding.
    Section 2.1 selects older films with limited IMDb visibility to reduce contamination; the generalization of results to modern films is assumed rather than demonstrated.
  • domain assumption The binary claim protocol yields more precise reasoning assessment than multiple-choice.
    The paper follows Karpinska et al. [17] and argues it reduces answer-ordering bias (Section 1); this is a design assumption, not a measured property.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding." pith.science (2026). https://pith.science/paper/RFJBCMUM

@misc{pith2026250606275,
  author       = {Pith},
  title        = {Pith review of: Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RFJBCMUM}},
  note         = {Machine review of arXiv:2506.06275}
}
abstract

Despite recent progress in vision-language models (VLMs), holistic understanding of long-form video content remains a significant challenge, partly due to limitations in current benchmarks. Many focus on peripheral, ``needle-in-a-haystack'' details, encouraging context-insensitive retrieval over deep comprehension. Others rely on large-scale, semi-automatically generated questions (often produced by language models themselves) that are easier for models to answer but fail to reflect genuine understanding. In this paper, we introduce MF$^2$, a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies (50-170 minutes long). MF$^2$ includes over 50 full-length, open-licensed movies, each paired with manually constructed sets of claim pairs -- one true (fact) and one plausible but false (fib), totalling over 850 pairs. These claims target core narrative elements such as character motivations and emotions, causal chains, and event order, and refer to memorable moments that humans can recall without rewatching the movie. Instead of multiple-choice formats, we adopt a binary claim evaluation protocol: for each pair, models must correctly identify both the true and false claims. This reduces biases like answer ordering and enables a more precise assessment of reasoning. Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance, underscoring the relative ease of the task for humans and their superior ability to retain and reason over critical narrative information -- an ability current VLMs lack.

Figures

Figures reproduced from arXiv: 2506.06275 by the authors.

Figure 1
Figure 1. Illustration of three claim pairs (each with a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Dataset construction process involving three main stages: movie collection, data annotation, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of claim pairs across reasoning granularities ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Pairwise accuracy for Gemini 2.5 Pro per comprehension dimension and reasoning granu [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Pairwise accuracy for large-scale models with video and subtitles, and human baseline per [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Guidelines provided for the data annotation procedure (Part 1). [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Guidelines provided for the data annotation procedure (Part 2). [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Guidelines provided for the data annotation procedure (Part 3). This part of the guidelines [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Guidelines provided for the data annotation procedure (Part 4). This part of the guidelines [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Guidelines provided for human evaluation (Part 1). [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Guidelines provided for human evaluation (Part 2). [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Direct prompt template used for open-weight models. Text in gray is optional, depending on the input modality used. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Explanation prompt template used for closed models. Text in gray is optional, depending on the input modality used. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

81 extracted references · 28 canonical work pages

  1. [1]

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen ...

    arXiv doi:10.48550/arxiv.2404.14219 2024

  2. [2]

    Infinibench: A comprehensive benchmark for large multimodal models in very long video understanding, 2024

    Kirolos Ataallah, Chenhui Gou, Eslam Abdelrahman, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. Infinibench: A comprehensive benchmark for large multimodal models in very long video understanding, 2024. URLhttps://arxiv.org/abs/2406.19875

    arXiv 2024

  3. [3]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  4. [4]

    Memory consolidation enables long-context video understanding

    Ivana Balazevic, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Henaff. Memory consolidation enables long-context video understanding. InForty-first International Conference on Machine Learning, 2024

    2024

  5. [5]

    Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks.arXiv preprint arXiv:2406.18403, 2024

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks.arXiv preprint arXiv:2406.18403, 2024

    arXiv 2024

  6. [6]

    Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding. InAdvances in Neural Information Processing Systems, volume 37, 2024. 10

    work page 2024

  7. [7]

    Cg-bench: Clue-grounded question answering benchmark for long video understanding.arXiv preprint arXiv:2412.12075, 2024

    Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg-bench: Clue-grounded question answering benchmark for long video understanding.arXiv preprint arXiv:2412.12075, 2024

    arXiv 2024

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024

    2024

Show all 81 references
  1. [9]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch,...

    2024 arXiv

  2. [10]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406.14515, 2024

    2024 arXiv

  3. [11]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever compre...

    2024

  4. [13]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

  5. [14]

    Movienet: A holistic dataset for movie understanding

    Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 709–727. Springer, 2020

    2020

  6. [15]

    Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks

    Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Emp...

    2023

  7. [16]

    Needle in a haystack - pressure testing LLMs, 2024

    Greg Kamradt. Needle in a haystack - pressure testing LLMs, 2024. URL https://github. com/gkamradt/LLMTest_NeedleInAHaystack

    2024

  8. [17]

    One thousand and one pairs: A “novel” challenge for long-context language models

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A “novel” challenge for long-context language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in ...

    2024 doi

  9. [18]

    TVQA: Localized, compositional video question answering

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. TVQA: Localized, compositional video question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p...

    2018 doi

  10. [19]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems,...

    2020

  11. [20]

    Merlot reserve: Neural script knowledge through vision and language and sound

    Amanpreet Li, Rowan Zellers, Youngjae Yu, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. InCVPR, 2022

    2022

  12. [21]

    Aria: An open multimodal native mixture-of-experts model, 2025

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, Chongyan Zhu, Xiaoyi Ren, Chao Li, Yifan Ye, Peng Liu, Lihuan Zhang, Hanshu Yan, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of...

    2025 arXiv

  13. [22]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  14. [23]

    Anchored answers: Unravelling positional bias in gpt-2’s multiple- choice questions.arXiv preprint arXiv:2405.03205, 2024

    Ruizhe Li and Yanjun Gao. Anchored answers: Unravelling positional bias in gpt-2’s multiple- choice questions.arXiv preprint arXiv:2405.03205, 2024

    2024 arXiv

  15. [24]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61s...

    2023 doi

  16. [25]

    Llama-vid: An image is worth 2 tokens in large language models, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023

    2023

  17. [26]

    World model on million-length video and language with blockwise ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=HN8V0flwJF

    2025

  18. [27]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  19. [28]

    Is your video language model a reliable judge? InThe Thirteenth International Conference on Learning Representations, 2025

    Ming Liu and Wensheng Zhang. Is your video language model a reliable judge? InThe Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview.net/forum?id=m8yby1JfbU

    2025

  20. [29]

    Nvila: Efficient frontier visual language models, 2024

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolon...

    2024

  21. [30]

    Addressing blind guessing: Calibra- tion of selection bias in multiple-choice question answering by video language models.arXiv preprint arXiv:2410.14248, 2024

    Olga Loginova, Oleksandr Bezrukov, and Alexey Kravets. Addressing blind guessing: Calibra- tion of selection bias in multiple-choice question answering by video language models.arXiv preprint arXiv:2410.14248, 2024

    2024 arXiv

  22. [31]

    Ovis: Structural Embedding Alignment for Multimodal Large Language Model.arXiv e-prints, art

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural Embedding Alignment for Multimodal Large Language Model.arXiv e-prints, art. arXiv:2405.20797, May 2024. doi: 10.48550/arXiv.2405.20797

    2024 doi arXiv

  23. [32]

    Valley: Video assistant with large language model enhanced ability, 2023

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability, 2023

    2023

  24. [33]

    Videoau- toarena: An automated arena for evaluating large multimodal models in video analysis through user simulation.arXiv preprint arXiv:2411.13281, 2024

    Ziyang Luo, Haoning Wu, Dongxu Li, Jing Ma, Mohan Kankanhalli, and Junnan Li. Videoau- toarena: An automated arena for evaluating large multimodal models in video analysis through user simulation.arXiv preprint arXiv:2411.13281, 2024. 12

    2024 arXiv

  25. [34]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024

    2024

  26. [35]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.arXiv preprint arXiv:2308.09126, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.arXiv preprint arXiv:2308.09126, 2023

    2023 arXiv

  27. [36]

    Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering.arXiv preprint arXiv:2503.14996, 2025

    Francesco Maria Molfese, Luca Moroni, Luca Gioffrè, Alessandro Scirè, Simone Conia, and Roberto Navigli. Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering.arXiv preprint arXiv:2503.14996, 2025

    2025 arXiv

  28. [37]

    Neptune: The long orbit to benchmarking long video understanding, 2025

    Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hornung, Nitesh Bharadwaj Gun- davarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, Mikhail Sirotenko, Yukun Zhu, and Tobias Weyand. Neptune: The long orbit to benchmarking long video understanding, 202...

    2025 arXiv

  29. [38]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex ...

    2024 arXiv

  30. [39]

    Movie plot analysis via turning point identification

    Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. Movie plot analysis via turning point identification. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In...

    2019 doi

  31. [40]

    Screenplay summariza- tion using latent narrative structure

    Pinelopi Papalampidi, Frank Keller, Lea Frermann, and Mirella Lapata. Screenplay summariza- tion using latent narrative structure. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors,Proceedings of the 58th Annual Meeting of the Association for Computati...

    1920 doi

  32. [41]

    Egoplan-bench2: A benchmark for multimodal large language model planning in real-world scenarios.arXiv preprint arXiv:2412.04447, 2024

    Lu Qiu, Yi Chen, Yuying Ge, Yixiao Ge, Ying Shan, and Xihui Liu. Egoplan-bench2: A benchmark for multimodal large language model planning in real-world scenarios.arXiv preprint arXiv:2412.04447, 2024

    2024 arXiv

  33. [42]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila an...

    2021

  34. [43]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  35. [44]

    Cinepile: A long video question answering dataset and benchmark.arXiv preprint arXiv:2405.08813, 2024

    Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark.arXiv preprint arXiv:2405.08813, 2024. 14

    2024 arXiv

  36. [45]

    McNamee, and André F

    Saul Santos, António Farinhas, Daniel C. McNamee, and André F. T. Martins. ∞-video: A training-free approach to long video understanding via continuous-time memory consolidation. arXiv preprint arXiv:2501.19098, 2025. URLhttps://arxiv.org/abs/2501.19098

    2025 arXiv

  37. [46]

    Trusting your evidence: Hallucinate less with context-aware decoding

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen- tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Ch...

    2024

  38. [47]

    It is too many options: Pitfalls of multiple-choice questions in generative ai and medical education

    Shrutika Singh, Anton Alyakin, Daniel Alexander Alber, Jaden Stryker, Ai Phuong S Tong, Karl Sangwon, Nicolas Goff, Mathew de la Paz, Miguel Hernandez-Rovira, Ki Yun Park, et al. It is too many options: Pitfalls of multiple-choice questions in generative ai and medical educati...

    2025 arXiv

  39. [48]

    Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    2023 arXiv

  40. [49]

    Moviechat+: Question-aware sparse memory for long video question answering.arXiv preprint arXiv:2404.17176, 2024

    Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering.arXiv preprint arXiv:2404.17176, 2024

    2024 arXiv

  41. [50]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  42. [51]

    Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- language ...

    2025 arXiv

  43. [52]

    AdaCAD: Adaptively decoding to balance conflicts between contextual and parametric knowledge

    Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. AdaCAD: Adaptively decoding to balance conflicts between contextual and parametric knowledge. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedings of the 2025 Conference of the Nations of the Americas ...

    2025

  44. [53]

    Multimodal needle in a haystack: Benchmarking long- context capability of multimodal large language models.arXiv preprint arXiv:2406.11230, 2024

    Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long- context capability of multimodal large language models.arXiv preprint arXiv:2406.11230, 2024

    2024 arXiv

  45. [54]

    Lvbench: An extreme long video understanding benchmark, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024

    2024

  46. [55]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. InEuropean Conference on Computer Vision, pages 58–76. Springer, 2024

    2024

  47. [56]

    Videollamb: Long-context video understanding with recurrent memory bridges, 2024

    Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long-context video understanding with recurrent memory bridges, 2024. URL https://arxiv.org/abs/2409. 01071

    2024

  48. [57]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209, 2024

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209, 2024. 15

    2024 arXiv

  49. [58]

    Tenenbaum, and Chuang Gan

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/fo...

    2021

  50. [59]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: //openreview.net/forum?...

    2024

  51. [60]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021

    2021

  52. [61]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  53. [62]

    Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024

    2024

  54. [63]

    Just ask: Learning to answer questions from millions of narrated videos

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. InICCV, 2021

    2021

  55. [64]

    Justice or prejudice? quantifying biases in LLM-as-a-judge

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in LLM-as-a-judge. InThe Thirteenth International Conference on Learning Re...

    2025

  56. [65]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. InAAAI, pages 9127–9134, 2019

    2019

  57. [66]

    Merlot: Multimodal neural script knowledge models

    Rowan Zellers, Ximing Lu, Youngjae Yu, Jae Sung Park, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. InNeurIPS, 2021

    2021

  58. [67]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding.arX...

    2025 arXiv

  59. [68]

    A simple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21715–21737, 2024

    2024

  60. [69]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023. URL https://arxiv.org/abs/2306.02858

    2023 arXiv

  61. [70]

    Movqa: A benchmark of versatile question-answering for long-form movie understanding, 2023

    Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding, 2023. URLhttps://arxiv.org/abs/2312.04817

    2023 arXiv

  62. [71]

    Video Instruction Tuning With Synthetic Data.arXiv e-prints, art

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video Instruction Tuning With Synthetic Data.arXiv e-prints, art. arXiv:2410.02713, October 2024. doi: 10.48550/arXiv.2410.02713

    2024 doi arXiv

  63. [72]

    Needle in a video haystack: A scalable synthetic evaluator for video MLLMs

    Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, weipeng chen, and Jing Liu. Needle in a video haystack: A scalable synthetic evaluator for video MLLMs. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://...

    2025

  64. [73]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  65. [74]

    Pool” of movies (the “Pool

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    1945 doi arXiv

  66. [78]

    The two claims should differ by minimal edits, meaning they should be as similar as possible while maintaining contrast

    Writing ClaimsYou are asked to create pairs of contrastive claims, where one claim is true (fact) and the counterfactual version is false (fib). The two claims should differ by minimal edits, meaning they should be as similar as possible while maintaining contrast. Each claim ...

  67. [79]

    Examples for Reasoning Granularity

    Select Claim GranularityFor each pair of claims you constructed, indicate whether answering them correctly requires reasoning based on a single scene, multiple scenes, or globally within the movie. Definition of scene: A scene in film refers to a complete unit of storytelling,...

  68. [80]

    Other" and suggest a new category. Note:The categorization is based on both claims (fact and fib). Check the examples provided in the “Examples for Comprehension Dimensions

    Claim CategorizationIdentify the comprehension dimensions the constructed pair of claims examines. Sometimes more than one dimension is examined, so we allow for multiple labels. Comprehension Dimension Labels: • Event/Entity Understanding:it refers to claims that require the ...

  69. [81]

    Pay attention to details and context in the movie, as some claims may be subtle or require careful reasoning

    Watch the entire movie carefully before proceeding to the evaluation. Pay attention to details and context in the movie, as some claims may be subtle or require careful reasoning

  70. [82]

    Start Classifying Claims

    After watching, it’s time to proceed to Stage 1.Please do not go back to the movie until Stage 1 of the test is completed.Press the “Start Classifying Claims” button, and you will be shown withone claim at a time. For each claim shown, you need to do the following: • Classify ...

  71. [2023]

    doi: 10.18653/v1/2023.emnlp-main.308

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.308. URLhttps://aclanthology.org/2023.emnlp-main.308/

    2023 doi

  72. [2024]

    doi: 10.18653/v1/2024.naacl-short.69

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-short.69. URL https://aclanthology.org/2024.naacl-short.69/

    2024 doi

  73. [2025]

    URLhttps://arxiv.org/abs/2501.13826

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.