Pith. sign in

REVIEW 2 major objections 5 minor 5 cited by

CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CausalVQA tests physical causal reasoning with 1,586 real-world video questions; the best model scores 61.66% where humans score 84.78%, with the largest gaps on anticipation and hypothetical questions.

desk verdict A serious, well-documented benchmark for physical causal reasoning in real video, but the headline model-human gap is overstated because paired model scores are compared to an unpaired human baseline. read the letter →

arxiv 2506.09943 v1 pith:IS4B7SZS submitted 2025-06-11 cs.CV cs.AI

Aaron Foss , Chloe Evans , Sasha Mitts , Koustuv Sinha , Ammar Rizvi , Justine T. Kao This is my paper
classification cs.CVcs.AI
keywords videoquestionansweringcausalreasoningphysicalcounterfactualhypotheticalanticipationegocentricmultimodalmodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a benchmark to test whether video AI can reason about cause and effect in the physical world, rather than only describe what is visible. CausalVQA consists of 1,586 multiple-choice questions derived from 298 egocentric videos of real activities such as basketball, rock climbing, and cooking, organized into five question types: counterfactual, hypothetical, anticipation, planning, and descriptive. The central finding is that current frontier multimodal models fall well short of humans: the best model evaluated, Gemini 2.5 Flash, scored 61.66% paired accuracy versus 84.78% for humans, with the largest gaps on anticipation and hypothetical questions. The benchmark was engineered through a multi-stage pipeline, including blind-LLM filtering, human quality assurance, and paired question perturbation, so that correct answers require visual understanding rather than linguistic shortcuts. If the finding stands, it indicates that models built for the physical world still lack the predictive and counterfactual reasoning that human users would rely on.

What carries the argument

The central mechanism is the paired-question design. Every question exists in two versions, MCQ and MCQ', whose distractor sets are LLM-paraphrased and reordered, and a model receives credit only if it answers both versions correctly (paired accuracy). This design penalizes guessing and sensitivity to answer phrasing and ordering, and a variance analysis in Appendix D argues that the paired benchmark has roughly the statistical power of an unpaired test four times its length. The pairing is supported by a creation pipeline: annotators generate questions from EgoExo4D clips; Llama 3.1-70B-Instruct refines language and removes hint words; a blind-LLM filter drops about 39% of items that a text-only model can answer without video; human QA repairs or removes flawed items; and agreement among 15 naive annotators per question assigns empirical difficulty (easy, medium, hard) and removes questions whose modal human answer disagrees with the ground truth label.

What would settle it

Run the same questions under a true paired human protocol: have the same people answer both MCQ and MCQ' for each question, score them with the same paired rule used for models, and compare that number with the reported 84.78%. If the resulting human paired accuracy lands near the model scores (61.66% or below), the paper's central claim of a large human-model gap on physical causal reasoning would be substantially weakened.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that state-of-the-art video models can describe visible events with moderate skill but cannot yet reliably predict physical outcomes, imagine alternatives, or reason about the consequences of actions in real-world settings. Humans scored 84.78% on the benchmark and 90.63% on descriptive questions, while the best model (Gemini 2.5 Flash) scored 61.66% overall and 76.04% on descriptive questions; on the reasoning subset (all types except descriptive), humans scored 81.43% against Gemini's 53.46%. The largest shortfalls appear on anticipation questions, where the average model scored 39.34% paired against 83.54% for humans, and hypothetical questions, where the average model scored 39.55% against 75.92% for humans. The paper interprets these results as evidence that current systems are not yet combining spatial-temporal perception, physical knowledge, and reasoning about alternatives at the level needed to act in the physical world.

Load-bearing premise

The headline gap assumes the human baseline of 84.78% is a paired score comparable to the models' 61.66%, but no annotator saw both versions of the same question (Section 3.1), so a true human paired score was never measured and would likely be lower.

Editorial extensions

If this is right

  • Video models are not yet reliable at anticipating physical outcomes in real-world settings, so applications on wearables or robots cannot yet depend on them for prediction and planning.
  • Descriptive skill does not transfer to causal reasoning: the best model's 76.04% on descriptive questions drops to 53.46% on reasoning questions, while humans drop only from 90.63% to 81.43%.
  • Anticipation and hypothetical questions are the clearest bottlenecks, with average model scores near 39% paired against human scores above 75%, so those question types are the ones to track for progress.
  • Every model tested scores 10-12% lower under paired scoring than unpaired, meaning answers remain unstable to phrasing and ordering; benchmark users should report paired accuracy rather than unpaired.
  • Because every item carries a human-derived difficulty label, future evaluations can check whether model gains come on questions humans also find hard, rather than only on easy ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the headline human-model gap is likely overstated, because the human baseline was not measured under the same paired scoring used for models — no annotator saw both versions of the same question (Section 3.1), and the human numbers in the paired and unpaired tables are identical. A true paired human score, requiring the same person to answer both MCQ and MCQ' correctly, was ne
  • Editorial inference: the blind-LLM filter appears model-specific — the paper reports that a different blind model (Qwen2.5-72B) still answers about 25% of items correctly without video, and the filtering model was not re-run after the later human QA stage, which could have re-introduced linguistic hints. A stronger blind model might therefore solve more items without seeing the video, shrinking th
  • Editorial inference: a concrete calibration would be to re-run the human protocol with the same annotators answering both versions, and to re-run blind filtering after human QA; both measurements would translate the raw 22-point gap into a comparable paired human score.
  • Editorial inference: since humans already reach 71.83% paired with only a single frame while the single-frame model scores 21.69%, the benchmark's challenge seems to come from the substance of the causal questions rather than from long-video memory, suggesting that frame-based or image-only variants could isolate reasoning ability from temporal perception.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper introduces CausalVQA, a video multiple-choice QA benchmark of 793 paired questions (1,586 items) built from egocentric EgoExo4D videos, covering five question types: descriptive, anticipation, planning, counterfactual, and hypothetical. The construction pipeline combines human question writing with LLM/VLM distractor generation, language refinement, paired answer perturbation, blind-LLM filtering for visual groundedness, human quality assurance, and human-derived difficulty labels. The authors evaluate six multimodal models zero-shot and report paired accuracies, with Gemini 2.5 Flash achieving the best score of 61.66% overall versus a human baseline of 84.78%, and with the largest gaps on anticipation and hypothetical questions.

Significance. If the reported gap is accurate, CausalVQA would be a valuable resource: it targets real-world physical causal reasoning rather than simulation or surface perception, incorporates several shortcut-mitigation mechanisms, and ships code, a debug dataset, a leaderboard, and a detailed datasheet. The paired-question design is a useful idea for increasing statistical power. However, the headline comparison is weakened by a metric mismatch between paired model scores and an unpaired human baseline, and the difficulty-level analysis is partly circular. With those issues resolved, the qualitative conclusion that frontier models still trail humans on physically grounded prediction and counterfactual reasoning is likely to survive, but the magnitude of the gap needs to be restated.

major comments (2)
  1. [§2.2.5, §3.1, Tables 6 and 12] The headline comparison mixes the paired metric for models with an unpaired human baseline. Section 3.1 states that 'no annotator saw both versions of the same question,' so a same-person paired human score is undefined, yet the Human column of the paired results table (Table 6) is identical to that of the unpaired table (Table 12), while every model's paired score is 10–12% lower than its unpaired score. Under independence, a paired human score would be roughly 72% (0.848^2), and it cannot exceed the per-version rate of 84.78%; the reported gap of more than 22 points (Gemini 2.5 Flash 61.66% vs. humans 84.78%) is therefore an apples-to-oranges comparison. The authors should either report the human baseline as unpaired and compare it with unpaired model scores (Gemini 2.5 Flash 71.63%, a gap of about 13 points), or collect and report a true paired human baseline; the quantitative claims in the abstract and §3.3 must be revised accordingly.
  2. [§2.2.8, Table 6] The empirical difficulty labels are computed from human agreement rates on the same items that are later used to assess models 'by difficulty' (easy >86.67% agreement, medium 70–86.67%, hard 36.67–70%). This makes the observation that models score much worse on hard items partly tautological: an item is hard precisely because the ground-truth answer is often not the modal human choice, so lower model accuracy on those items is expected. Please either provide an independent difficulty measure (e.g., item statistics not derived from the same human responses) or reframe the analysis as 'performance on items with high vs. low human disagreement,' and tone down the interpretation in §3.3 that this reflects a distinct model-specific deficiency.
minor comments (5)
  1. [Abstract] The abstract states the dataset contains 1,786 items, but §2.3.1 and the datasheet report 1,586 items (793 paired questions); please correct the count.
  2. [§2.2.6 and Appendix C.2] The residual blind-LLM accuracy of about 25% (Qwen2.5-72B-Instruct) is near chance and largely supports the filter's effectiveness, but the wording that questions 'require' deep visual understanding is too strong; consider stating explicitly that a small fraction of items remain answerable from text alone.
  3. [Appendix D] The variance-reduction argument assumes that a guessing model's responses to MCQ and MCQ' are independent; please state this assumption explicitly and, if possible, verify it empirically with the collected model response logs rather than presenting the factor-of-four claim as a mathematical certainty.
  4. [§3.1 and Table 6] Clarify in the text and in the Table 6 caption that the human baseline is an unpaired, per-version accuracy pooled over both versions, not a paired score; this will prevent future readers from making the same apples-to-oranges comparison.
  5. [Tables 6 and 12] There are small formatting errors in the table captions ('except fordescriptive' should read 'except for descriptive'), and Table 7's dot-glyph cells make the comparison difficult to read; consider using standard symbols and a legend.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark’s construction and the model–human comparison rest on independent measurements, and the apparent circularity concerns (difficulty levels, blind-LLM filtering) are not reductions of predictions to inputs.

full rationale

The paper’s central claim—that frontier multimodal models underperform humans on CausalVQA—is based on separately measured human and model accuracies. Human difficulty labels are derived from human agreement (Section 2.2.8), but model scores are computed independently on the same items; observing that models perform worse on items humans find hard is an empirical correlation, not a tautology, because the model scores are not used to define the labels. The blind-LLM filter (Section 2.2.6) removes items answerable by Llama 3.1-70B without video; the paper then validates this with a different blind model (Qwen2.5-72B, Appendix C.2), finding near-chance performance, which is an external check rather than a circular justification. The paired-versus-unpaired human baseline mismatch (Tables 6 and 12) is a statistical comparability concern, not a circularity: the human number is an input and the model numbers are inputs, and the gap is simple arithmetic rather than a derived quantity that reduces to its own definition. No load-bearing self-citations, imported uniqueness theorems, or renamed known results appear; the paper also discloses residual shortcut sensitivity in its Limitations section. Overall, the derivation chain is not circular.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The benchmark's central claims rest on human consensus as ground truth, on a particular operationalization of visual groundedness (a blind LLM cannot answer), and on the paired-question variance model. These are domain assumptions rather than standard math. The difficulty thresholds are hand-chosen free parameters. There are no fitted physical parameters and no invented physical entities.

free parameters (2)
  • Difficulty agreement thresholds = easy >86.67%, medium 70-86.67%, hard 36.67-70%
    Hand-chosen cutoffs applied to human agreement rates; they define the easy, medium, and hard labels used throughout the model analysis (Section 2.2.8).
  • Annotators per item = 15
    The authors chose 15 naive annotators per question version for the human baseline and difficulty assignment; this sample size determines the granularity of agreement rates but is not derived from a formal power analysis (Section 2.2.8).
assumptions (6)
  • domain assumption Naive human consensus defines the correctness of answers to physical causal reasoning questions.
    Section 2.2.8 accepts items only when the modal response of 15 annotators matches the annotator-provided ground truth and removes 14% of items otherwise. The benchmark's ground truth is therefore human agreement, not an externally verified physical ground truth.
  • domain assumption A question is visually grounded if a text-only LLM cannot answer both MCQ versions correctly.
    Section 2.2.6 filters roughly 39% of items using Llama 3.1-70B-Instruct. Appendix C.2 acknowledges residual shortcut susceptibility, so this operationalization is approximate.
  • domain assumption The five-question taxonomy (counterfactual, hypothetical, anticipation, planning, descriptive) is a valid decomposition of physical causal reasoning.
    Section 2.2.2 defines the taxonomy and uses it to guide annotation. The benchmark's construct validity depends on this taxonomy being a fair representation of causal reasoning in physical settings.
  • domain assumption The paired-question variance reduction assumes the unknown portion of the benchmark is answered at chance independently across versions.
    Appendix D derives an effective 4x increase in test size by treating unknown items as pure guessing (p=0.2 per version) with no covariance. Correlated model errors across MCQ and MCQ' would weaken this statistical benefit.
  • domain assumption The manually filtered EgoExo4D clips are representative of diverse real-world physical causality.
    Section 2.2.1 selects 298 egocentric videos spanning sports, cooking, and bike repair. The real-world relevance claim depends on this selection being sufficiently diverse and representative.
  • domain assumption The LLM language-refinement steps preserve question meaning.
    Section 2.2.4 uses automated LLM checks for meaning preservation but does not report a quantitative validation of semantic equivalence. If meaning drifted systematically, both model and human performance could be affected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models." pith.science (2026). https://pith.science/paper/IS4B7SZS

@misc{pith2026250609943,
  author       = {Pith},
  title        = {Pith review of: CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IS4B7SZS}},
  note         = {Machine review of arXiv:2506.09943}
}
read the original abstract

We introduce CausalVQA, a benchmark dataset for video question answering (VQA) composed of question-answer pairs that probe models' understanding of causality in the physical world. Existing VQA benchmarks either tend to focus on surface perceptual understanding of real-world videos, or on narrow physical reasoning questions created using simulation environments. CausalVQA fills an important gap by presenting challenging questions that are grounded in real-world scenarios, while focusing on models' ability to predict the likely outcomes of different actions and events through five question types: counterfactual, hypothetical, anticipation, planning and descriptive. We designed quality control mechanisms that prevent models from exploiting trivial shortcuts, requiring models to base their answers on deep visual understanding instead of linguistic cues. We find that current frontier multimodal models fall substantially below human performance on the benchmark, especially on anticipation and hypothetical questions. This highlights a challenge for current systems to leverage spatial-temporal reasoning, understanding of physical principles, and comprehension of possible alternatives to make accurate predictions in real-world settings.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RetroHolmes: When Semantic Plausibility Fails Retrospective Physical Process Reasoning

    cs.MM 2026-07 conditional novelty 7.0 of 10

    Current VLMs fail retrospective physical reachability and causal reconstruction on RetroHolmes; a simple analysis-by-synthesis loop with video simulation reduces bias and belief-conflict sensitivity.

  2. ChainReaction: Causal Chain-Guided Reasoning for Modular and Explainable Causal-Why Video Question Answering

    cs.CV 2025-08 conditional novelty 7.0 of 10

    ChainReaction splits causal-why video QA into a chain extractor and a chain-driven answerer, and reports SOTA or near-SOTA accuracy with interpretable intermediate causal chains.

  3. SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

    cs.CV 2025-11 conditional novelty 6.0 of 10

    SWITCH introduces a 193-video benchmark of tangible control-interface interactions and shows that frontier LMMMs struggle with fine-grained grounding and outcome verification.

  4. OpenWorldLib: A Unified Codebase and Definition of Advanced World Models

    cs.CV 2026-04 unverdicted novelty 4.0 of 10

    OpenWorldLib offers a standardized codebase and definition for world models that combine perception, interaction, and memory to understand and predict the world.

  5. Embodied AI Agents: Modeling the World

    cs.AI 2025-06 conditional novelty 4.0 of 10

    Embodied AI agents should be built around physical world models plus a mental world model of the user, with virtual, wearable, and robotic agents sharing this core.

Reference graph

Works this paper leans on

104 extracted references · 50 canonical work pages · cited by 5 Pith papers

  1. [1]

    Social-iq: A question answering benchmark for artificial social intelligence

    Amir Zadeh 0001, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8807–8817. Computer Vision Foundation / IEEE, 2019. URLhttp://openaccess.t...

    2019

  2. [2]

    Sheng, Wei Emma Zhang, Munazza Zaib, and Ahoud Alhazmi

    Elaf Alhazmi, Quan Z. Sheng, Wei Emma Zhang, Munazza Zaib, and Ahoud Alhazmi. Distractor generation in multiple-choice tasks: A survey of methods, datasets, and evaluation, 2024. URLhttps://arxiv.org/abs/2402. 01512

    2024

  3. [3]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2...

    arXiv 2025

  4. [4]

    Phyre: A new benchmark for physical reasoning, 2019

    Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning, 2019. URLhttps://arxiv.org/abs/1908.05656

    arXiv 2019

  5. [5]

    Videophy: Evaluating physical commonsense for video generation, 2024

    Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsense for video generation, 2024. URLhttps://arxiv.org/abs/2406.03520

    arXiv 2024

  6. [6]

    Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video-language understanding, 2024. URL https://arxiv.org/abs/2411.04998

    arXiv 2024

  7. [7]

    Tenenbaum, and Chuang Gan

    Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B. Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos, 2022. URLhttps://arxiv.org/abs/ 2205.01089

    arXiv 2022

  8. [8]

    Perceptionlm: Open-access data and models for detailed visual understanding

    Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Hanoona Rasheed, Peize Sun, Po-Yao Huang, Daniel Bolya, Suyog Jain, Miguel Martin, Huiyu Wang, Nikhila Ravi, Shashank Jain, Temmy Stark, Shane Moon, Babak Damavandi, Vivian Lee, Andrew Westbury, Salman Khan, Philipp ...

    arXiv 2025

Show all 104 references
  1. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    Zhe Chen et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URLhttps://arxiv.org/abs/2412.05271

    2025 arXiv

  2. [10]

    Datasheets for datasets, 2021

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets, 2021. URLhttps://arxiv.org/abs/1803.09010

    2021 arXiv

  3. [11]

    Causal reasoning through intervention.Causal learning: Psychology, philosophy, and computation, 5, 2007

    York Hagmayer, Steven A Sloman, David A Lagnado, and Michael R Waldmann. Causal reasoning through intervention.Causal learning: Psychology, philosophy, and computation, 5, 2007

    2007

  4. [12]

    Shot2story: A new benchmark for comprehensive understanding of multi-shot videos, 2025

    Mingfei Han, Linjie Yang, Xiaojun Chang, Lina Yao, and Heng Wang. Shot2story: A new benchmark for comprehensive understanding of multi-shot videos, 2025. URLhttps://arxiv.org/abs/2312.10300

    2025 arXiv

  5. [13]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  6. [14]

    The kinetics human action video dataset, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017. URLhttps://arxiv.org/abs/1705.06950

    2017 arXiv

  7. [15]

    Processing counterfactual and hypothetical conditionals: An fmri investigation.NeuroImage, 72:265–271, 2013

    Eugenia Kulakova, Markus Aichhorn, Matthias Schurz, Martin Kronbichler, and Josef Perner. Processing counterfactual and hypothetical conditionals: An fmri investigation.NeuroImage, 72:265–271, 2013

    2013

  8. [16]

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. Tvqa: Localized, compositional video question answering,

  9. [17]

    Lmms-eval: Accelerating the development of large multimoal models, March 2024

    Bo Li*, Peiyuan Zhang*, Kaichen Zhang*, Fanyi Pu*, Yuhao Dong Xinrun Du, Haotian Liu, Yuanhan Zhanga, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024. URLhttps://github.com/EvolvingLMMs-Lab/lmms-eval

    2024

  10. [18]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URLhttps://arxiv.org/abs/ 2408.03326

    2024 arXiv

  11. [19]

    A survey on benchmarks of multimodal large language models, 2024

    Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang. A survey on benchmarks of multimodal large language models, 2024. URLhttps://arxiv.org/abs/2408.08632

    2024 arXiv

  12. [20]

    From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022

    2022

  13. [21]

    From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering, 2022

    Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering, 2022. URLhttps://arxiv.org/abs/2205.14895

    2022 arXiv

  14. [22]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra,...

    2024

  15. [23]

    Egoschema: A diagnostic benchmark for very long-form video language understanding, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding, 2023. URLhttps://arxiv.org/abs/2308.09126

    2023 arXiv

  16. [24]

    Towards world simulator: Crafting physical commonsense-based benchmark for video generation,

    Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense-based benchmark for video generation,

  17. [25]

    Do generative video models understand physical principles?arXiv preprint arXiv:2501.09038, 2025

    Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles?arXiv preprint arXiv:2501.09038, 2025

    2025 arXiv

  18. [26]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023. URLhttps://openai.com/index/gpt-4v-system-card/

    2023

  19. [27]

    Gpt-4o, 2024

    OpenAI. Gpt-4o, 2024. URLhttps://openai.com/index/gpt-4o-system-card/

    2024

  20. [28]

    CRIPP-VQA: Counterfactual reasoning about implicit physical properties via video question answering

    Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. CRIPP-VQA: Counterfactual reasoning about implicit physical properties via video question answering. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022

    2022

  21. [29]

    Cripp-vqa: Counterfactual reasoning about implicit physical properties via video question answering, 2022

    Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Cripp-vqa: Counterfactual reasoning about implicit physical properties via video question answering, 2022. URLhttps://arxiv.org/abs/2211.03779

    2022 arXiv

  22. [30]

    Large language models sensitivity to the order of options in multiple-choice questions, 2023

    Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions, 2023. URLhttps://arxiv.org/abs/2308.11483

    2023 arXiv

  23. [31]

    Perception test: A diagnostic benchmark for multimodal video models, 2023

    Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koste...

    2023 arXiv

  24. [32]

    Cinepile: A long video question answering dataset and benchmark, 2024

    Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark, 2024. URLhttps://arxiv.org/abs/2405.08813

    2024 arXiv

  25. [33]

    Intphys: A framework and benchmark for visual intuitive physics reasoning, 2020

    Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning, 2020. URL https://arxiv.org/abs/1803.07616

    2020 arXiv

  26. [34]

    Ego4d goal-step: Toward hierarchical understanding of procedural activities

    Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Informat...

    2023

  27. [35]

    Ego4d: Around the world in 3,000 hours of egocentric video, 2022

    Ego4D Team. Ego4d: Around the world in 3,000 hours of egocentric video, 2022. URLhttps://arxiv.org/abs/ 2110.07058

    2022 arXiv

  28. [36]

    Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives,

    EgoExo4D Team. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives,

  29. [37]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530

    2024 arXiv

  30. [38]

    Gemini 2.5 flash, 2025

    Gemini Team. Gemini 2.5 flash, 2025. URLhttps://deepmind.google/technologies/gemini/flash/

    2025

  31. [39]

    URLhttps://arxiv.org/abs/2311.18259

    arXiv

  32. [40]

    Weischedel, and Nanyun Peng

    Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Reddy Chandra, Marjorie Freedman, Ralph M. Weischedel, and Nanyun Peng. Acquired: A dataset for answering counterfactual questions in real-life videos, 2023. URL https://arxiv.org/abs/2311.01620

    2023 arXiv

  33. [41]

    Next-qa:next phase of question-answering to explaining temporal actions, 2021

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question-answering to explaining temporal actions, 2021. URLhttps://arxiv.org/abs/2105.08276

    2021 arXiv

  34. [42]

    The llama 3 herd of models, 2024

    Llama Team. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783

    2024 arXiv

  35. [43]

    Activitynet-qa: A dataset for understanding complex web videos via question answering, 2019

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering, 2019. URLhttps://arxiv.org/abs/1906.02467

    2019 arXiv

  36. [44]

    Mm-llms: Recent advances in multimodal large language models, 2024

    Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models, 2024. URLhttps://arxiv.org/abs/2401.13601

    2024 arXiv

  37. [45]

    Tenenbaum

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning, 2020. URLhttps://arxiv.org/abs/1910.01442

    2020 arXiv

  38. [46]

    distractors

    Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, and Tat-Seng Chua. Video question answering: Datasets, algorithms and challenges, 2022. URLhttps://arxiv.org/abs/2203.01225. 15 Appendix DataSheet CausalVQA benchmark datasheet, following the framework introduced by G...

    2022 arXiv

  39. [48]

    Contphy: Continuum physical concept learning and reasoning from videos

    Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua B Tenenbaum, and Chuang Gan. Contphy: Continuum physical concept learning and reasoning from videos. InInternational Conference on Machine Learning. PMLR, 2024

    2024

  40. [50]

    Annotator watches the video clip (segmented at the start/end time stamps provided by Phase 1 annotators)

  41. [51]

    Reads the question and set of choices (without seeing which one is the correct answer)

  42. [52]

    Attempts to select the correct answer

  43. [53]

    Tool reveals the correct answer

  44. [54]

    Descriptive ii

    Answers the following questions (more than one item may be wrong - several of them may need to be flagged): (a) According to the definitions, what type of question is this? i. Descriptive ii. Anticipation iii. Planning iv. Counterfactual v. Hypothetical (b) Is there sufficient...

  45. [55]

    A census-matched pool of age and gender for hired evaluators

  46. [56]

    Multiple questions included solely to filter for both attention and task understanding

  47. [57]

    To account for variance in the length of videos associated with each question, we standardize the amount of time an evaluator will spend across questions by assigning a set of videos with an approximately equal absolute duration per evaluator

  48. [58]

    Evaluators are never shown both possible questions for a video, to eliminate the influence of within- question bias and memory on selected answers

  49. [59]

    your answer

    Task descriptions given to evaluators emphasize in multiple ways the goal of e.g. predicting a sequence or end state. Our goal was to ensure strong understanding by the evaluators (even though none had prior exposure to the benchmark), such that the baseline would more closely...

    2025

  50. [61]

    Carefully observe the Clip given with the prompt

  51. [63]

    The answers should extremely plausible, but incorrect, based on what you see in the Clip

    Devise a list of four Distractors that are extremely plausible and could be correct answers to the Revised Question. The answers should extremely plausible, but incorrect, based on what you see in the Clip . They could be correct given the information in the Scene Description ...

  52. [64]

    Distractors **must** be similar in length and complexity to the Revised Answer

    Check to make the Distractors are similar to, but differentiated from the Revised Answer. Distractors **must** be similar in length and complexity to the Revised Answer. Distractors **must** have unique meanings when compared to each other and the Revised Answer, but rely on s...

  53. [65]

    Revise the Distractors as needed unil you have your list of four

    Ensure that the Distractors make sense and are extremely plausible as answers to the Revised Question given what you saw in the Clip, even though they are incorrect. Revise the Distractors as needed unil you have your list of four

  54. [66]

    Modify the Distractors as needed to ensure they are all different from the Revised Answer

    Check to make certain you have not included the Revised Answer (or its equivalent) among the Distractors. Modify the Distractors as needed to ensure they are all different from the Revised Answer

  55. [67]

    [Distractor 1| Distractor 2| etc...]

    Supply the rationale for your Distractors. Responses must follow this format specifically: —– Distractors: "[Distractor 1| Distractor 2| etc...]" END Rationale: "the rationale for the Distractors." END —– You often send the Revised Answer in the Distractors. Follow these steps...

  56. [68]

    Before you output the Distractors, check to see if the Revised Answer is present in the Distractors (remember, they are separated by |)

  57. [69]

    If the Revised Answer is present, remove it from the Distractors and generate another Distractor that is different from the Revised answer and the remaining Distractors, but similar in length, language, and complexity

  58. [70]

    [’distractor 1.’| ’distractor 2.’| ’distractor 3.’| ’distractor 4.’]

    Return the new Distractors and Rationale in the correct format. —– Here is an example of correct format: Distractors: "[’distractor 1.’| ’distractor 2.’| ’distractor 3.’| ’distractor 4.’]" Rationale: "The original answer is correct because...The other answers are incorrect bec...

  59. [74]

    You follow the following steps in this task:

    The person will turn to the left and then to the left again 25 B.2 Language Refinement Prompt for LLM to refine the questions and answer optons You are an expert at understanding questions and editing them to use common sense language. You follow the following steps in this task:

  60. [75]

    Carefully read the Original Question and the Original Answer

  61. [76]

    Please sound as polished as possible and make certain to capture the gist of the Original Answer

    Give a one sentence rewritten version of the answer that improves it by using simple, concise (substitute one word for two, if possible), common sense, idiomatically correct American English in active voice. Please sound as polished as possible and make certain to capture the ...

  62. [77]

    Check that the rewritten answer makes sense for the question

  63. [78]

    Ensure the rewritten answer captures the gist of the Original Answer

  64. [79]

    Ensure that language in the question reflects the language in the rewritten answer

    Next, try to rewrite the question using normal, common sense, idiomatically correct American English in active voice. Ensure that language in the question reflects the language in the rewritten answer

  65. [80]

    Now make certain that the rewritten question has the same meaning as the Original Question

    Revise the rewritten question and substitute any words that may provide a hint at the answer (e.g., specific nouns or verbs) with generic/neutral ones. Now make certain that the rewritten question has the same meaning as the Original Question

  66. [81]

    the rewritten answer you came up with in step 2, adjusted for any changes from steps 3 and 4

    Supply your rationale for the changes you make in Answer Reasons and Question Reasons, respectively. Responses will follow this format: —– A.Rewritten Answer: "the rewritten answer you came up with in step 2, adjusted for any changes from steps 3 and 4." END B.Rewritten Questi...

  67. [82]

    Carefully read the Original Answer and the New Answer

  68. [83]

    Assess whether or not the New Answer is very similar to the Original Answer. Do they have the same essential meaning? If the New Answer adds information not present in the Original Answer that changes the meaning, correct the New Answer so that captures the gist of the Origina...

  69. [84]

    Next, carefully read the Original Question and the New Question

  70. [85]

    the rewritten New Answer you came up with in step 2

    Assess whether or not the New Question is very similar to the Original Question. Do they have the same essential meaning? If the New Question adds information not present in the Original Question or changes the meaning, correct the New Question so that it captures the gist of ...

  71. [86]

    Important Words will only be found in the Original Question

    Carefully read the Original Question and the Original Answer and assess if any Important Words in the Original Question give a hint as to what the Original Answer may be. Important Words will only be found in the Original Question. There can be any number of Important Words in...

  72. [87]

    Carefully read the AI First Pass Answer and the Original Answer and give the similarity a Score from 0 (not similar) to 2 (essentially the same)

  73. [88]

    Important Words will only be found in the Original Question

    If your Score is equal to or greater than 1, assess if any Important Words in the Original Question give a hint as to what the AI First Pass Answer may be. Important Words will only be found in the Original Question. Cross check it against your Key Words list and add any new I...

  74. [89]

    defensive position

    Revise the Original Question into a Rewritten Question that substitutes the Key Words for neutral/generic words you can think of so that it does not provide a hint as to the Original Answer or AI First Pass Answer, but would keep the question’s meaning extremely similar. Be ca...

  75. [90]

    Ensure that the Original Answer would still make sense as a correct response to the Rewritten Question

  76. [91]

    Fix the Rewritten Question using highly generic and neutral language that maintains the Original Question’s meaning so that the Original Answer will make sense

  77. [92]

    the answer you came up with in step 1, 2, 3, 4, and 6

    Supply the rationale for your changes. Responses must follow this format specifically: —– A. Rewritten Question: "the answer you came up with in step 1, 2, 3, 4, and 6." END B. Similarity Score: "the Score from step 2." END C. Key Words: "[Important Word 1, Important Word 2, e...

  78. [93]

    Carefully read and understand the Revised Question and the Revised Answer

  79. [94]

    |", ",",

    Carefully read and understand the Possibilities. These are potential answers to the Revised Question that are often incorrect. They are a list of four sentences (usually separated by "|", ",", "." or " ") and are sometimes prefaced by "Distractors: ". These have been generated...

  80. [95]

    Carefully read and understand the Scene Description and the Next Scene Description

  81. [96]

    They **must** be similar in length and complexity to the Revised Answer

    Devise a list of four New Distractors that are polished versions of the remaining Possibilities and that are plausible and could be correct answers to the Revised Question. They **must** be similar in length and complexity to the Revised Answer. The answers should be extremely...

  82. [97]

    Encourage yourself to think outside the box and consider unconventional explanations or scenarios

    Check to make sure the New Distractors are not only **completely** plausible but also **divergent** from each other and from the Revised Answer. Encourage yourself to think outside the box and consider unconventional explanations or scenarios. Double check that you are thinkin...

  83. [98]

    **Never** repeate the Revised Answer in the New Distractors

    Ensure that the New Distractors make sense as answers to the Revised Question in the same context, even though they are incorrect. **Never** repeate the Revised Answer in the New Distractors. **Never** repeate any of the individual elements in the New Distractors list. Revise ...

  84. [99]

    Supply the rationale for your New Distractors

  85. [100]

    [New Distractor 1| New Distractor 2| etc...]

    Generate a second list of Synonym Distractors that substitute common sense (not overly stilted) synonyms for some words in the Distractors (you’ll want to change more than one, generally), but keeps the meaning of the Distractors the same. Responses must follow this format spe...

  86. [101]

    The person will turn to the left next

  87. [102]

    The person will turn to the left and then to the right

  88. [103]

    The person will turn to the right and then to the left

  89. [104]

    The person will move backward to create space B

    The person will turn to the left and then to the left again •The person will advance to strike the ball •The person will retreat to gain distance • The person will pivot to the left to evade a challenger •The person will leap up to meet the ball 29 Table 11Example showing the ...

  90. [105]

    It is supplied as a string of numbered answers separated by a pipe |

    Read the list of Possible Answers. It is supplied as a string of numbered answers separated by a pipe |

  91. [106]

    You must choose an answer, even if you are not sure

    Choose Your Answer from the Possible Answers – pick the one that is most correct. You must choose an answer, even if you are not sure

  92. [107]

    Your Answer from the Possible Answers as a string

    Return Your Answer as a single string. Record Your Answer Number. Supply the Rationale for your answer. Responses must follow this format specifically: —– A. Your Answer: "Your Answer from the Possible Answers as a string" END B. Your Answer Number: "The number at the beginnin...

  93. [108]

    binomial varianceσ 2 is a function ofp(probability correct) σ2 =p(1−p) and,

  94. [109]

    It is easy to showd is reduced by pairing, as it is entirely dependent on the ratio of variances between an unpaired and paired benchmark

    overall variance from two sources is: σ2 =w 2 1σ2 1 +w 2 2σ2 2 + 2w1w2cov(σ1, σ2) substituting here: w1 =kthe known fraction w2 =uthe unknown fraction σ2 1 =σ 2 k =p k(1−p k) = 100%(1−100%) = 0 σ2 2 =σ 2 upu(1−p u)wherep u is the chance/guess rate cov(σ1, σ2) = 0sinceσ 1 = 0an...

  95. [2019]

    URLhttps://arxiv.org/abs/1809.01696. 13

    arXiv

  96. [2024]

    URLhttps://arxiv.org/abs/2410.05363

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.