Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The VideoMathQA benchmark claims that real math reasoning over video requires integrating visual, audio, and text cues across time, and that today's best models manage only 44.8 percent accuracy.

desk verdict A genuinely new and carefully annotated video-math benchmark, but the paper never verifies its central claim that the questions cannot be solved from a single frame or transcript. read the letter →

arxiv 2506.05349 v2 pith:2VV5CAQB submitted 2025-06-05 cs.CV

Hanoona Rasheed , Abdelrahman Shaker , Anqi Tang , Muhammad Maaz , Ming-Hsuan Yang , Salman Khan , Fahad Shahbaz Khan This is my paper
classification cs.CV
keywords videoquestionansweringmultimodalmathematicalreasoningtemporalbenchmarkconstructionchain-of-thoughtevaluationlargelanguagemodelseducationalvideos
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces VideoMathQA, a benchmark of 420 real-world video-question pairs designed to test whether AI models can carry out mathematical reasoning that spans time and modalities. Its central claim is that the questions cannot be solved from a static frame, a transcript, or a single image; the model must integrate evolving diagrams, handwritten or on-screen notation, and spoken cues distributed non-linearly across videos that run from ten seconds to over an hour. To support diagnosis, every question carries four to ten expert-annotated reasoning steps, totaling 2,945 steps. The evaluation finds a large gap in current capability: the best tested model, GPT-o4-mini, reaches 44.8 percent accuracy in the chain-of-thought multi-binary setting with subtitles. A sympathetic reader would care because the benchmark offers a way to tell whether a model reasons across multimodal time or merely perceives.

What carries the argument

The central object is the 'needle-in-a-multimodal-haystack' task design: each video-question pair is selected so the answer requires locating and integrating sparse cues across high-resolution frames, handwritten or digital text, and spoken narration over time. Three mechanisms carry the evaluation: a three-stage expert annotation pipeline that filters videos for rich temporal progression and writes step-by-step solution traces; a multi-binary evaluation, which pairs the correct multiple-choice answer against each distractor so guessing is penalized; and a chain-of-thought step evaluation that scores model rationales against the 2,945 expert steps, using an LLM judge, to detect confabulated reasoning and diagnose error categories.

What would settle it

Run the full set of 420 questions under strict single-frame-only and transcript-only conditions, using strong vision-language and text-only models; if either condition reaches accuracy near the best video-model scores (around 44.8 percent), the benchmark's claim that temporal cross-modal integration is necessary would be falsified.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that a dedicated benchmark built around the 'needle-in-a-multimodal-haystack' problem—finding and combining sparse, essential cues scattered across visual, textual, and audio streams over time—exposes a clear capability gap in current multimodal models. The benchmark spans ten mathematical domains and three reasoning types: direct problem solving, conceptual transfer, and deep instructional comprehension over partially worked solutions. Across thirty evaluated models, accuracy is consistently low, with the strongest result being GPT-o4-mini at 44.8 percent under chain-of-thought multi-binary evaluation with subtitles; open-source leaders such as Qwen2.5-VL-72B reach roughly 28 to 29 percent in the same setting. The paper also reports that subtitles help larger reasoning-capable models, that performance drops on long videos, and that the dominant failure mode is question misunderstanding rather than computation or formula selection.

Load-bearing premise

The load-bearing premise is that the 420 questions genuinely require integrating information across time and modalities, and cannot be answered reliably from a static frame, a transcript, or a single image; this property is asserted through expert video-selection judgment, not demonstrated by a quantitative leakage test on the full set.

Editorial extensions

If this is right

  • If the benchmark measures what it claims, model rankings on VideoMathQA reflect genuine temporal cross-modal reasoning rather than static perception or single-frame reading.
  • The gap between top proprietary and open-source models is narrowing, with optimized open models already surpassing older proprietary systems on this benchmark.
  • Subtitles and dense frame sampling improve performance mainly for larger, reasoning-capable models, implying that future video-math systems should be built to exploit both modalities jointly.
  • Step-wise reasoning scores allow a fine-grained diagnosis of where reasoning fails, which could guide targeted training data and evaluation beyond final-answer accuracy.
  • Long-video questions remain the hardest, so progress on VideoMathQA likely requires advances in long-context integration and sustained attention, not just larger parameter counts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct leakage test on the full set—single-frame-only and transcript-only input—would settle whether the temporal-cross-modal premise holds, since the paper currently rests it on annotator judgment.
  • The 2,945 expert reasoning steps could double as instruction-tuning data; a testable extension is whether fine-tuning on the traces lifts model accuracy on the same style of question.
  • Running human experts on the same 420 questions would provide a ceiling and reveal whether the benchmark's difficulty comes from true reasoning depth or from quirks of the selected videos.
  • The multi-binary and step-level scoring protocol transfers naturally to other temporally extended multimodal benchmarks, not just mathematics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. VideoMathQA is a benchmark of 420 video-question pairs drawn from instructional mathematics videos, annotated by graduate-level experts with multi-step reasoning traces (2,945 steps total) across ten mathematical domains and three reasoning types: direct problem solving, conceptual transfer, and deep instructional comprehension. The paper evaluates 30 proprietary and open-source multimodal models under multiple-choice (MCQ) and multi-binary (MBin) settings, with and without subtitles, using both direct answering and chain-of-thought prompting, plus a frame-sampling ablation and an LLM-based step-wise evaluation and error analysis. The headline results are that the best model, GPT-o4-mini, achieves 44.8% accuracy in the CoT MBin setting with subtitles, that open-source models are narrowing the gap with proprietary ones, and that models struggle most on long videos and on questions requiring integration of sparse cross-modal cues.

Significance. If the benchmark's construct validity is established, VideoMathQA fills a genuine gap: existing mathematical reasoning benchmarks are largely static-image or text-based, and existing video benchmarks do not provide step-level mathematical reasoning annotations. The paper's strengths include the broad evaluation across 30 models, the release of evaluation code in LMMs-Eval, the expert annotation effort, the reasoning-type taxonomy, the subtitle and frame-sampling ablations, and the attempt to diagnose reasoning failures through step-wise evaluation and error categories. However, the central claim that the benchmark isolates temporally extended cross-modal reasoning rather than static perception or transcript reading is not yet supported by an upper-bound leakage test; the automatic step judge is not validated against human ratings; and the random baseline in the main table is inconsistent with chance levels. These issues are fixable within the scope of the manuscript, but they are load-bearing for the paper's central claims and must be addressed before the benchmark can be fully trusted.

major comments (3)
  1. [§3.2, Fig. 5a] The benchmark's central premise—that questions cannot be reliably answered from static frames or audio transcripts alone—is asserted through annotator judgment but is never tested as an upper bound. The vision-blind text-only and single-image comparisons in Fig. 5a are lower bounds: the text-only model is not given the same subtitles used in the +Sub condition, and the single-image model is tested on an author-selected key frame rather than on a per-question best-frame or oracle-frame selection over the full 420 questions. Consequently, if a transcript-only model with the aligned subtitles or a best-frame image-only model reached video-condition accuracy, the benchmark would measure perception or text reading rather than temporal cross-modal integration. Please add a leakage test on the full benchmark: (i) a text-only LLM supplied with the exact aligned subtitles used in the +Sub condition, and (ii) image-only evaluation with per-question best/oracle frame selection, and report both against the video-condition numbers.
  2. [§4.1(iv), Table 2] The step-wise CoT evaluation and the associated error analysis both rely on Qwen3-4B as an automatic judge, but no validation against human expert ratings is reported. The paper presents CoT Eval scores (e.g., 6.9 for GPT-o4-mini) and the Fig. 5c error distribution as diagnostic results; without a human agreement study (e.g., correlation with expert scores or inter-annotator agreement on a sample), these numbers are uncalibrated and may reflect the judge's own biases rather than the quality of model reasoning. Please add a validation study comparing Qwen3-4B step scores and error labels with expert annotations on a random subset of responses.
  3. [Table 1] The random baseline row is inconsistent with chance levels. For five-choice MCQ, random accuracy should be 20%, and for the multi-binary protocol (four independent binary pairs per question), random accuracy should be 6.25%, yet Table 1 reports 17.4% and 7.9%, respectively. Unless a different protocol, tie-breaking rule, or empirical sampling procedure is being used, this should be corrected and explained; as printed, it distorts the calibration of all model scores and makes it harder to interpret the significance of the gap between models and chance.
minor comments (5)
  1. [§3.2] There is a duplicated phrase: 'the videos review the videos and select those' should be 'the videos and select those'.
  2. [References] References [32] and [33] are the same CLEVRER entry, and references [9] and [10] are the same Video-MMMU entry; please merge the duplicates.
  3. [Table 1] The table header 'Mathematic Concepts' should be 'Mathematical Concepts' or 'Mathematics Concepts'.
  4. [General] The paper would benefit from a human expert accuracy estimate on a sample of the benchmark; without it, the headline accuracy of 44.8% lacks an interpretable ceiling and the claimed difficulty levels are harder to calibrate.
  5. [§4.1] The text inconsistently refers to 'LMMs-Eval' and 'lmms_eval'; please standardize the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: VideoMathQA is an annotation and evaluation benchmark whose claims are empirical measurements, not derivations that reduce to their inputs.

full rationale

VideoMathQA does not present a derivation chain from first principles; it constructs a benchmark and reports model accuracy. The load-bearing premise that questions require temporally extended cross-modal reasoning is enforced during annotation (Section 3.2: videos are selected so questions 'cannot be reliably answered using static frames or audio transcripts alone'), but this is an unverified construct-validity assumption, not a circular reduction: no equation, fitted parameter, or self-citation is later used to define or force a benchmark result. Evaluations are external measurements on 30 models, and the same authors running their own benchmark is normal practice rather than load-bearing self-citation. The LLM-as-judge for step evaluation is a methodological choice, not a derivation that assumes the conclusion. Although a rigorous leakage test (image-only or transcript-only upper bounds) is absent and would strengthen validity, the absence of that test is a limitation, not circularity under the stated criteria. No specific step can be quoted where a claimed prediction is equivalent by construction to the benchmark's own inputs, so the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; VideoMathQA is a dataset artifact, not a postulated theoretical entity. The listed free parameters are author-chosen design thresholds, and the axioms are unverified assumptions about the benchmark's validity and the evaluation pipeline.

free parameters (3)
  • Video duration buckets = short <30s, medium 30s-2min, long >2min
    Hand-chosen thresholds to group results; no external basis, and they affect the reported duration trend.
  • Difficulty levels = easy 13%, medium 30%, hard 57%
    Assigned by expert annotators; the distribution shapes the claim that models fail hardest on hard questions.
  • Step count range = 4 to 10 steps per question
    Arbitrary range chosen by authors; affects the granularity of step evaluation.
assumptions (3)
  • domain assumption The 420 questions truly require temporal cross-modal reasoning and cannot be solved from static frames or transcripts alone.
    Assumed in Section 3.2 based on annotator judgment; no quantitative leakage test is reported.
  • domain assumption The Qwen3-4B LLM judge reliably scores model reasoning against ground-truth steps.
    Used for step evaluation in Section 4.1; no human agreement or correlation analysis is provided.
  • domain assumption Multiple-choice accuracy (MCQ and MBin) is a valid proxy for mathematical reasoning ability.
    Standard assumption in QA benchmarks; the paper does not justify it beyond reproducibility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos." pith.science (2026). https://pith.science/paper/2VV5CAQB

@misc{pith2026250605349,
  author       = {Pith},
  title        = {Pith review of: VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2VV5CAQB}},
  note         = {Machine review of arXiv:2506.05349}
}
abstract

Mathematical reasoning in real-world video settings presents a fundamentally different challenge than in static images or text. It requires interpreting fine-grained visual information, accurately reading handwritten or digital text, and integrating spoken cues, often dispersed non-linearly over time. In such multimodal contexts, success hinges not just on perception, but on selectively identifying and integrating the right contextual details from a rich and noisy stream of content. To this end, we introduce VideoMathQA, a benchmark designed to evaluate whether models can perform such temporally extended cross-modal reasoning on videos. The benchmark spans 10 diverse mathematical domains, covering videos ranging from 10 seconds to over 1 hour. It requires models to interpret structured visual content, understand instructional narratives, and jointly ground concepts across visual, audio, and textual modalities. We employ graduate-level experts to ensure high quality, totaling over $920$ man-hours of annotation. To reflect real-world scenarios, questions are designed around three core reasoning challenges: direct problem solving, where answers are grounded in the presented question; conceptual transfer, which requires applying learned methods to new problems; and deep instructional comprehension, involving multi-step reasoning over extended explanations and partially worked-out solutions. Each question includes multi-step reasoning annotations, enabling fine-grained diagnosis of model capabilities. Through this benchmark, we highlight the limitations of existing approaches and establish a systematic evaluation framework for models that must reason, rather than merely perceive, across temporally extended and modality-rich mathematical problem settings. Our benchmark and evaluation code are available at: https://mbzuai-oryx.github.io/VideoMathQA

Figures

Figures reproduced from arXiv: 2506.05349 by the authors.

Figure 1
Figure 1. The foundation of our benchmark is the “needle-in-a-multimodal-haystack” challenge, capturing the core difficulty of cross-modal reasoning across time from visual, textual, and audio streams. Built on this, VideoMathQA categorizes each question along four key dimensions: reasoning type, mathematical concept, video duration, and difficulty. arXiv:2506.05349v2 [cs.CV] 24 Jun 2025 [PITH_FULL_IMAGE:figures/full_fig_p00… view at source ↗
Figure 2
Figure 2. Example questions from the VIDEOMATHQA benchmark illustrating the three reasoning types: Problem Focused, Concept Transfer, and Deep Comprehension. The benchmark includes evolving dynamics in a video, complex text prompts, five multiple-choice options, the expert-annotated step-by-step reasoning to solve the given problem, and the final correct answer as shown above. Existing benchmarks for mathematical reasoning (e… view at source ↗
Figure 3
Figure 3. The figure illustrates a) Distribution of questions and model performance across ten mathematical concepts in the VIDEOMATHQA. The consistently low performance across all concepts reveals a significant gap in the ability of the current multimodal models to perform mathematical reasoning over videos. b) Distribution of video durations in VIDEOMATHQA, highlighting a diverse range from short clips of 10s to long-videos… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The figure shows VIDEOMATHQA performance a) Across video duration categories using the CoT MBin +Sub setting; b) Impact of subtitles under the CoT MBin setting; and c) Effect of varying the number of input frames under CoT MCQ setting. Overall, models perform best on m…
Figure 5
Figure 5. Figure 5: The figure shows a) Comparison among vision-blind, image-only, and video models, highlighting the need for video-level understanding to perform well in VIDEOMATHQA. b) Distribution of questions in VIDEOMATHQAacross three difficulty levels for varying reasoning depths, …

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.

  2. VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A multi-tool video agent trained with dual-branch RL (BiSPO) and a new VideoSearch-QA benchmark substantially beats open-source agentic baselines on video deep research and multimodal search tasks.

Reference graph

Works this paper leans on

41 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    MathQA: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini et al. MathQA: Towards interpretable math word problem solving with operation-based formalisms. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019

    work page 2019

  3. [3]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

    arXiv 2025

  4. [4]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  5. [5]

    Perceptionlm: Open-access data and models for detailed visual understanding.arXiv preprint arXiv:2504.13180, 2025

    Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muham- mad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al. Perceptionlm: Open-access data and models for detailed visual understanding.arXiv preprint arXiv:2504.13180, 2025

    arXiv 2025

  6. [6]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  7. [7]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    arXiv 2025

  8. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025

    work page 2025

Show all 41 references
  1. [10]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.arXiv preprint arXiv:2501.13826, 2025

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  2. [11]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems ...

    2023

  3. [12]

    Lmms-eval: Accelerating the development of large multimodal models, 2024

    Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, 2024. URLhttps://github.com/EvolvingLMMs-Lab/lmms-eval

    2024

  4. [13]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  5. [14]

    Aria: An open multimodal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024

    2024 arXiv

  6. [15]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195–22206...

    2024

  7. [16]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  8. [17]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  9. [18]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    2024 arXiv

  10. [19]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, et al. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InICLR, 2024

    2024

  11. [20]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  12. [21]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 2024

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 2024

    2024

  13. [22]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the Association for Computational Linguistics: ACL 2022, 2022

    2022

  14. [23]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean et al. Perception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems, 2023

    2023

  15. [24]

    Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models

    Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. arXiv preprint arXiv:2410.23266, 2024

    arXiv

  16. [25]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  17. [26]

    Qwen3, April 2025

    Qwen Team. Qwen3, April 2025. URLhttps://qwenlm.github.io/blog/qwen3/

    2025

  18. [27]

    Measuring multimodal mathematical reasoning with the math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with the math-vision dataset. InAdvances in Neural Information Processing Systems, 2024

    2024

  19. [28]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025

    2025 arXiv

  20. [29]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. InAdvances in Neural Information Processing Systems, 2024

    2024

  21. [30]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777–9786, 2021

    2021

  22. [31]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13204–13214, 2024

    2024

  23. [33]

    Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenen- baum. Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019. 16

    1910 arXiv

  24. [34]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. InProceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127–9134, 2019

    2019

  25. [35]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI

    Xiang Yue et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. InCVPR, 2024

    2024

  26. [36]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024

    2024 arXiv

  27. [37]

    Lmms-eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URLhttps://arxiv.org/abs/2407.12772

    2024 arXiv

  28. [38]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  29. [39]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  30. [40]

    Worldqa: Multimodal world knowledge in videos through long-chain reasoning.arXiv preprint arXiv:2405.03272, 2024

    Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christopher Arif Setiadharma, Jingkang Yang, and Ziwei Liu. Worldqa: Multimodal world knowledge in videos through long-chain reasoning.arXiv preprint arXiv:2405.03272, 2024

    2024 arXiv

  31. [41]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  32. [42]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  33. [43]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. InICLR, 2025. 17

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.