Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HumanPCR shows that today's multimodal models fail at human-centric video reasoning because they rely on question cues instead of proactively seeking visual evidence.

desk verdict A serious benchmark effort with a genuinely new proactive-evidence design, but the headline Human-R failure rates rest on unreleased annotation validity and need human baselines before they should be taken at face value. read the letter →

arxiv 2508.13692 v1 pith:JLKVFM2Y submitted 2025-08-19 cs.CV

Keliang Li , Hongze Shen , Hao Shi , Ruibing Hou , Hong Chang , Jie Huang , Chenghao Jia , Wen Wang
show 4 more authors
Yiling Wu Dongmei Jiang Shiguang Shan Xilin Chen
This is my paper
classification cs.CV
keywords multimodallargelanguagemodelshuman-centricvideounderstandingproactivevisualevidencereasoningbenchmarkchain-of-thoughtrationalequery-guidedretrievalperceptioncomprehensionevaluationsuite
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HumanPCR, a three-level benchmark for how well multimodal large language models (MLLMs) understand people in real-world scenes: perception (Human-P), comprehension (Human-C), and reasoning (Human-R). Its central claim is that current models, including the strongest proprietary ones, systematically fail at human-centric video reasoning when the question does not point to the relevant visual evidence: the best model reaches only about 59% on Human-R, and most models score below 40%. Error analysis attributes most failures to missed 'proactive visual evidence' — information present in the video but not cued by the question — rather than to missing world knowledge. The paper argues that scaling frames, adding context-extraction or retrieval modules, and increasing test-time thinking produce only limited gains, because these methods are themselves query-guided. A sympathetic reader should care because the result separates genuine video understanding from retrieval of question-cued content, which is closer to what real-world use demands.

What carries the argument

The load-bearing device is the Human-R protocol and its construct of 'proactive visual evidence' — video information not, or only partially, indicated by the question. Each question passed a three-part meta-review: it must require multiple visual evidence, its evidence-integration pattern must not be fully determined by the question, and at least one essential evidence must be proactive. Human-annotated Chain-of-Thought rationales itemize the key visual evidence, making the extraction step observable rather than hidden. This construct does the work: it converts a vague 'models don't understand humans' claim into a documented failure to extract evidence outside the query's cue, and lets the p

What would settle it

Independent human baseline: have a panel of expert annotators answer the 442 Human-R questions without seeing the curated answers, and measure agreement with the ground truth (and with each other). If human agreement is well below ceiling or human accuracy is not near-perfect, the benchmark's answerability premise fails. A complementary check: re-word each question so the key proactive evidence is explicitly cued; if models then answer at near-ceiling accuracy, the deficit is question phrasing rather than a fundamental inability to seek evidence.

Watch

Extended reading notes

Core claim

Discovery: MLLMs fail at human-centric video reasoning because they retrieve evidence the question cues and miss 'proactive' evidence not cued by the question, so they never assemble the full human picture. Human-R measures this: 442 open-ended questions, each meta-reviewed to require multiple visual evidence with at least one essential piece unmentioned in the question, each with human-annotated CoT rationales. o4-mini scores 58.60%, o3 59.28%, most open-source models below 40%; error analysis shows missed proactive evidence dominates while referred evidence is rarely missed. Frame scaling, context extraction, best-of-n, and self-refinement give small or negative gains. The 6,176-question P

Load-bearing premise

Everything hinges on the 442 Human-R questions having one right answer that human experts would independently agree on; the paper reports a strict annotation-review pipeline but no inter-annotator agreement scores or human-baseline accuracy, so if many questions are ambiguous, the reported failure rates and 'missed proactive evidence' error counts partly measure benchmark noise.

Editorial extensions

If this is right

  • If the proactive-evidence finding holds, accuracy on Human-R is a more realistic estimate of real-world video understanding than accuracy on benchmarks whose questions name the relevant objects, because real queries rarely specify where to look.
  • The benchmark separates model families: open-source models match proprietary ones on perception and comprehension but fall behind on reasoning, so the open-source gap is specifically a multi-evidence integration problem.
  • Scaling model size continues to improve Human-R while perception and comprehension plateau near 38B parameters, suggesting that reasoning over multiple visual evidence is where scale still buys performance.
  • Context-extraction methods validated on Video-MME — frame selection, pooling, token selection, memory retrieval — show smaller or negative gains on Human-R, indicating that query-guided compression is not a route to holistic video understanding.
  • Test-time compute helps only through wide sampling: Best-of-N gives more than 5% gains on three base models, while Self-Refine saturates or degrades, and the strongest thinking model still leaves a wide gap to human-level reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension not in the paper: use the Human-R CoT rationales as supervision for an explicit evidence-selection step; if accuracy rises, the correlation between missed proactive evidence and failures becomes a causal demonstration.
  • The proactivity criterion likely transfers beyond human scenes: any video-QA benchmark that names the relevant objects in the question invites query-guided shortcuts, so Human-R-style meta-review could sharpen general long-video and world-model evaluation.
  • The model-specific error patterns (one model misses less proactive evidence but adds more irrelevant evidence; another is more selective) imply a precision-recall tradeoff in evidence extraction that could be scored automatically against the annotated rationales.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes HumanPCR, a three-level benchmark for human-centric multimodal understanding: Human-P (perception, 17 tasks), Human-C (comprehension, 17 tasks), and Human-R (442 manually curated open-ended video-reasoning questions with human-annotated chain-of-thought rationales). The authors evaluate over 30 open-source and proprietary MLLMs under a common protocol, reporting that most models score below 60% on Human-P/C and below 40% on Human-R, with the best model (o4-mini) reaching 58.60%. The central interpretive claim is that MLLMs fail on Human-R because they miss proactive visual evidence not cued by the question and rely on query-guided retrieval; the paper supports this with frame-scaling experiments, context-extraction comparisons, test-time-compute scaling, and a 200-question error taxonomy.

Significance. If the measurement is valid, HumanPCR would be a useful and fairly comprehensive diagnostic for human-centric MLLM evaluation. The three-level taxonomy, the 34-task coverage, the human-verified multiple-choice component, the human-annotated CoT rationales, and the extensive model sweep are all strengths. The Human-R design—requiring multiple visual evidence and proactive evidence seeking—targets a real gap in current video QA benchmarks, and the frame-scaling and test-time-compute experiments are informative. However, the headline diagnostic claim about missed proactive evidence depends entirely on the validity of the Human-R ground truth and its error labels. The current preprint does not establish inter-annotator agreement, a human expert baseline, or public access to the dataset, so the core measurement is not yet independently auditable. The direction of the result is plausible, but the magnitude and the error taxonomy are not yet verifiable.

major comments (4)
  1. [§3.2.3, Table 2] Human-R is the load-bearing instrument for the paper's central claim, but its measurement validity is not established. The manuscript reports a multi-stage QC pipeline and a below-1/5 acceptance rate, yet no inter-annotator agreement is given for the 442 Human-R items, no human expert baseline accuracy is reported, and the data are not released. With only 442 questions, a modest share of ambiguous or contestable items would materially shift the reported percentages (e.g., 22 ambiguous items would move o4-mini's 58.60% by roughly 5 points) and could inflate the 'missed proactive evidence' error category, since that label presupposes a unique gold answer and evidence chain. Please report IAA on a held-out subset, provide a human baseline, and release the videos, questions, answers, evidence chains, and CoTs for independent audit.
  2. [§4.1, Appendix D] Open-ended Human-R accuracy is adjudicated by o3-mini, but there is no validation that this judge agrees with human experts. If the judge is too strict, accepts plausible but wrong paraphrases, or is itself biased toward certain reasoning styles, all Human-R scores are systematically confounded. This directly affects the headline numbers in Table 2 and the comparisons in Table 3. Provide a human-scored subset with agreement statistics (e.g., Cohen's kappa) between o3-mini and expert annotators, and report sensitivity to the adjudication prompt or to using alternative strong judges.
  3. [§4.4, Figure 10] The error analysis is the direct evidence for the claim that models rely on query-guided retrieval and miss proactive visual evidence. However, the classification of 200 errors into five categories plus three subcategories is manual, with no stated protocol, no dual-annotation reliability, and no release of the classified examples. Because the labels 'Missed Proactive', 'Missed Referred', and 'Irrelevant Evidence' presuppose the exact evidence chain in the gold CoT, the taxonomy is only as valid as the gold evidence annotations. Please report inter-annotator agreement on error labels and make the 200 error instances and their classifications public.
  4. [§3.1, Figures 2 and 5] The definitions of 'proactive visual evidence' and the scoring rubric for evidence count are described only qualitatively. Since Human-R inclusion thresholds require at least one essential proactive evidence and multiple evidence, and since Figures 2 and 5 quantify proactive vs. referred evidence, the rubric itself should be operationalized. Please release the annotation checklist, item-level evidence scores, and examples that were accepted/rejected at meta-review, so readers can assess the construct validity of the proactive-evidence measure.
minor comments (4)
  1. [Throughout] The manuscript contains several typos and grammatical slips: 'artifical' (Abstract), 'includeing' (§3.1), 'suffient' (§3.1), 'levles' (§3.3), and 'specially' (Introduction). A careful proofread is needed.
  2. [Figure 10] The x-axis label in Figure 10(a) appears garbled ('/glyph1197umber'), and the panel captions are hard to parse. Please fix the rendering and clarify whether the bars show counts or percentages.
  3. [Table 3] The column structure of Table 3 is confusing: 'Direct', 'M=0 (CoT)', 'M=1', 'M=2', 'M=3' are not aligned consistently across model rows. Please unify the formatting and define exactly what each column measures.
  4. [§1] The sentence 'These results validate the necessity of our benchmark' is partly a restatement of the design choices (Human-R was built to require proactive multi-evidence reasoning). I recommend softening this to avoid the appearance of circularity: the results demonstrate that models fail on this constructed operationalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's construction filters are design choices, not fitted inputs; the evaluation results are empirical measurements independent of the authors' prior work.

full rationale

HumanPCR's central claim—that MLLMs miss proactive visual evidence and rely on query-guided retrieval—is supported by direct evaluations on 30+ models. The Human-R curation criteria (meta-reviewers require multiple visual evidence and at least one essential proactive evidence) are selection criteria for test items, not fitted parameters or derived predictions; they do not force the measured failure rates (best model 58.60% on Human-R, most below 40%). The error analysis in §4.4 is a manual classification of 200 questions, so its reliability is a measurement-validity concern (no inter-annotator agreement or human baseline is reported), but this is distinct from circularity: the labels do not appear in the benchmark construction equations. The sole self-citation [68] appears in Related Work as one item in a list of human-centric probing work and is not load-bearing. Conclusion's stated limitations (LLM-based metrics, academic dataset reliance) are acknowledged confounds, not circular steps. No equation or fitted input is reused as a 'prediction,' so no circularity is established.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The ledger shows the benchmark's contribution is real but rests on several unquantified human-judgment premises: the correctness and uniqueness of human-curated answers, the reliability of the proactive-evidence categorization, the validity of LLM adjudication, and the accuracy of source dataset labels. No formal or quantitative support (inter-annotator agreement, human baselines) is provided for these premises. There are no fitted numeric constants in the usual sense; the hand-set rubrics act as the free parameters.

free parameters (2)
  • Evidence-count scoring rubric (Figures 1, 2, 5)
    The number of 'pieces of visual evidence' a benchmark question demands is scored by the authors' manual analysis; the coding rules are not given in the main text and no inter-rater reliability is reported.
  • Human-R inclusion thresholds (multi-evidence, at least one essential proactive evidence) = acceptance rate below 1/5
    The meta-reviewer checklist operationalizes 'sufficient visual complexity'; the operational threshold is hand-set and its reliability is unquantified.
assumptions (5)
  • domain assumption Multiple-choice and open-ended QA accuracy measures the target human-centric visual understanding abilities
    The entire evaluation rests on treating QA accuracy as a valid proxy for perception, comprehension, and reasoning competence; standard in the field but not independently validated here.
  • domain assumption Human annotators and reviewers reliably judge question quality, answer correctness, and evidence presence
    The QC pipeline (§3.2.3) assumes reviewers can consistently identify proactive versus referred evidence and reject ambiguous questions; no agreement statistics are provided.
  • domain assumption The o3-mini adjudicator yields valid correctness labels for open-ended answers
    Used for all Human-R scoring (§4.1 Metrics); the authors acknowledge this as a limitation in the Conclusion.
  • domain assumption Source dataset annotations used to template Human-P and Human-C questions are correct
    P and C questions leverage existing annotations (§3.2.2); errors in source labels propagate into benchmark items despite human verification.
  • ad hoc to paper 'Proactive visual evidence' is a well-defined, meaningful construct for reasoning difficulty
    The construct is defined, illustrated, and operationalized by this paper's own protocol; it is a contribution but not an externally validated metric.
invented entities (1)
  • Proactive visual evidence (construct) independent evidence
    purpose: Core difficulty criterion and diagnostic category for Human-R; used to filter questions and to classify model errors
    Not a physical entity, but a new measurable construct. The paper gives concrete examples and the human CoT rationales let external readers audit the categorization, so it has a falsifiable handle. Inter-rater reliability is nonetheless unreported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes." pith.science (2026). https://pith.science/paper/JLKVFM2Y

@misc{pith2026250813692,
  author       = {Pith},
  title        = {Pith review of: HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JLKVFM2Y}},
  note         = {Machine review of arXiv:2508.13692}
}
read the original abstract

The aspiration for artificial general intelligence, fueled by the rapid progress of multimodal models, demands human-comparable performance across diverse environments. We propose HumanPCR, an evaluation suite for probing MLLMs' capacity about human-related visual contexts across three hierarchical levels: Perception, Comprehension, and Reasoning (denoted by Human-P, Human-C, and Human-R, respectively). Human-P and Human-C feature over 6,000 human-verified multiple choice questions, assessing massive tasks of 9 dimensions, including but not limited to essential skills frequently overlooked by existing benchmarks. Human-R offers a challenging manually curated video reasoning test that requires integrating multiple visual evidences, proactively extracting context beyond question cues, and applying human-like expertise. Each question includes human-annotated Chain-of-Thought (CoT) rationales with key visual evidence to support further research. Extensive evaluations on over 30 state-of-the-art models exhibit significant challenges in human-centric visual understanding, particularly in tasks involving detailed space perception, temporal understanding, and mind modeling. Moreover, analysis of Human-R reveals the struggle of models in extracting essential proactive visual evidence from diverse human scenes and their faulty reliance on query-guided retrieval. Even with advanced techniques like scaling visual contexts and test-time thinking yield only limited benefits. We hope HumanPCR and our findings will advance the development, evaluation, and human-centric application of multimodal models.

Figures

Figures reproduced from arXiv: 2508.13692 by the authors.

Figure 1
Figure 1. Overview of HumanPCR. (Left–upper) The hierarchical taxonomy of HumanPCR. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of Human-R and Video-MME examples. The distribution of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Benchmark Construction. A hierarchical task taxonomy and a diverse human-centric vision [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: The distribution of the visual evidence number. 3.2.1 Task Definition and Data Collection Protocol We adopt a task-driven data collection approach, sourcing from diverse curated datasets and internet videos, as illustrated in the top panel of [PITH_FULL_IMAGE:figures/…
Figure 6
Figure 6. Figure 6: The relationship between model size and macro-average accuracy. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Ablation study examining the impact of CoT. (a) Relative improvements of CoT of models [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Effect of frame sampling on Human-R. (a) Overall accuracy. (b–d) Accuracy grouped by [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Comparisons of advanced video understanding methods on Human-R and Video-MME[ [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Distribution of error types on Human-R across leading models (“f” means the number of [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Structure of hypernucleus $_{\Lambda}^{7}$Li within microscopic three-cluster model

    nucl-th 2025-08 unverdicted novelty 5.0 of 10

    The three-cluster model for 7Li-Lambda predicts narrow resonance states at 0-2 MeV above the 4He+d+Lambda threshold.

Reference graph

Works this paper leans on

104 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    arXiv 2024

  2. [2]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024

    arXiv 2024

  3. [3]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  4. [4]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  7. [7]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems , 36:62991–63010, 2023

    2023

  8. [8]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    arXiv 2009

Show all 104 references
  1. [9]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  2. [10]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision ...

    2022

  3. [11]

    Ego-exo4d: Understand- ing skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understand- ing skilled human activity from first-and third-person perspectives. In Proceedin...

    2024

  4. [12]

    Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments

    Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese. Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments. IEEE transactions on pattern analysis ...

    2021

  5. [13]

    Jrdb-social: A multifaceted robotic dataset for understanding of context and dynamics of human interactions within social groups

    Simindokht Jahangard, Zhixi Cai, Shiki Wen, and Hamid Rezatofighi. Jrdb-social: A multifaceted robotic dataset for understanding of context and dynamics of human interactions within social groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  6. [14]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015

    2015

  7. [15]

    Humanbench: Towards general human-centric perception with projector assisted pretraining

    Shixiang Tang, Cheng Chen, Qingsong Xie, Meilin Chen, Yizhou Wang, Yuanzheng Ci, Lei Bai, Feng Zhu, Haiyang Yang, Li Yi, et al. Humanbench: Towards general human-centric perception with projector assisted pretraining. In Proceedings of the IEEE/CVF Conference on Computer Visio...

    2023

  8. [16]

    Unihcp: A unified model for human-centric perceptions

    Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihcp: A unified model for human-centric perceptions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 178...

    2023

  9. [17]

    Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models

    Wenyi Hong, Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang. Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models. arXiv preprint arXiv:2501.02955, 2025

    2025 arXiv

  10. [18]

    Face-human-bench: A comprehensive benchmark of face and human understanding for multi-modal assistants

    Lixiong Qin, Shilong Ou, Miaoxuan Zhang, Jiangning Wei, Yuhang Zhang, Xiaoshuai Song, Yuchen Liu, Mei Wang, and Weiran Xu. Face-human-bench: A comprehensive benchmark of face and human understanding for multi-modal assistants. arXiv preprint arXiv:2501.01243, 2025

    2025

  11. [19]

    Humanvbench: Exploring human-centric video understanding capabilities of mllms with synthetic benchmark data

    Ting Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding, Yaliang Li, and Ying Shen. Humanvbench: Exploring human-centric video understanding capabilities of mllms with synthetic benchmark data. arXiv preprint arXiv:2412.17574, 2024

    2024 arXiv

  12. [20]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  13. [21]

    Actionatlas: A videoqa benchmark for domain-specialized action recognition

    Mohammadreza Reza Salehi, Jae Sung Park, Aditya Kusupati, Ranjay Krishna, Yejin Choi, Hanna Hajishirzi, and Ali Farhadi. Actionatlas: A videoqa benchmark for domain-specialized action recognition. Advances in Neural Information Processing Systems, 37:137372–137402, 2024

    2024

  14. [22]

    Posegpt: Chatting about 3d human pose

    Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Posegpt: Chatting about 3d human pose. CoRR, 2023

    2023

  15. [23]

    Personalized large vision-language models

    Chau Pham, Hoang Phan, David Doermann, and Yunjie Tian. Personalized large vision-language models. arXiv preprint arXiv:2412.17610, 2024

    2024 arXiv

  16. [24]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  17. [25]

    Mmbench: Benchmarking end-to-end multi-modal dnns and understanding their hardware-software implications

    Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tianhao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tongqiao Xu, et al. Mmbench: Benchmarking end-to-end multi-modal dnns and understanding their hardware-software implications. In 2023 IEEE International Symposium on Workl...

    2023

  18. [26]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006, 2024

    2024 arXiv

  19. [27]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  20. [28]

    Gaze360: Physically unconstrained gaze estimation in the wild

    Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Matusik, and Antonio Torralba. Gaze360: Physically unconstrained gaze estimation in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6912–6921, 2019

    2019

  21. [29]

    Posescript: 3d human poses from natural language

    Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. Posescript: 3d human poses from natural language. In European Conference on Computer Vision, pages 346–362. Springer, 2022. 13

    2022

  22. [30]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil...

    2025 arXiv

  23. [31]

    Introducing OpenAI o3 and o4-mini

    OpenAI. Introducing OpenAI o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/ , 2025

    2025

  24. [32]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  25. [33]

    Mmvu: Measuring expert-level multi-discipline video understanding

    Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Measuring expert-level multi-discipline video understanding. arXiv preprint arXiv:2501.12380, 2025

    2025 arXiv

  26. [34]

    Mmr-v: What’s left unsaid? a benchmark for multimodal deep reasoning in videos

    Kejian Zhu, Zhuoran Jin, Hongbang Yuan, Jiachun Li, Shangqing Tu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Mmr-v: What’s left unsaid? a benchmark for multimodal deep reasoning in videos. arXiv preprint arXiv:2506.04141, 2025

    2025 arXiv

  27. [35]

    Video- mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video- mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  28. [36]

    M3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv preprint arXiv:2405.16473, 2024

    2024 arXiv

  29. [37]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    2023 arXiv

  30. [38]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  31. [39]

    Morse-500: A programmatically controllable video benchmark to stress-test multimodal reasoning

    Zikui Cai, Andrew Wang, Anirudh Satheesh, Ankit Nakhawa, Hyunwoo Jae, Keenan Powell, Minghui Liu, Neel Jay, Sungbin Oh, Xiyao Wang, et al. Morse-500: A programmatically controllable video benchmark to stress-test multimodal reasoning. arXiv preprint arXiv:2506.05523, 2025

    2025 arXiv

  32. [40]

    Video-holmes: Can mllm think like holmes for complex video reasoning? arXiv preprint arXiv:2505.21374, 2025

    Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can mllm think like holmes for complex video reasoning? arXiv preprint arXiv:2505.21374, 2025

    2025 arXiv

  33. [41]

    Worldqa: Multimodal world knowledge in videos through long-chain reasoning.arXiv preprint arXiv:2405.03272, 2024

    Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christopher Arif Setiadharma, Jingkang Yang, and Ziwei Liu. Worldqa: Multimodal world knowledge in videos through long-chain reasoning.arXiv preprint arXiv:2405.03272, 2024

    2024 arXiv

  34. [42]

    Egonormia: Benchmarking physical social norm understanding

    MohammadHossein Rezaei, Yicheng Fu, Phil Cuvin, Caleb Ziems, Yanzhe Zhang, Hao Zhu, and Diyi Yang. Egonormia: Benchmarking physical social norm understanding. arXiv preprint arXiv:2502.20490, 2025

    2025 arXiv

  35. [43]

    Mmtom-qa: Multimodal theory of mind question answering

    Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. Mmtom-qa: Multimodal theory of mind question answering. arXiv preprint arXiv:2401.08743, 2024

    2024 arXiv

  36. [44]

    Vcr-bench: A comprehensive evaluation framework for video chain-of- thought reasoning

    Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao. Vcr-bench: A comprehensive evaluation framework for video chain-of- thought reasoning. arXiv preprint arXiv:2504.07956, 2025

    2025 arXiv

  37. [45]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  38. [46]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  39. [47]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Information Processing Systems, 37:28828– 28857, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Information Processing Systems, 37:28828– 28857, 2024. 14

    2024

  40. [48]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093, 2024

    2024

  41. [49]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024

    2024 arXiv

  42. [50]

    Adaretake: Adaptive redun- dancy reduction to perceive longer for video-language understanding

    Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redun- dancy reduction to perceive longer for video-language understanding. arXiv preprint arXiv:2503.12559, 2025

    2025 arXiv

  43. [51]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  44. [52]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    2023

  45. [53]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  46. [54]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  47. [55]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  48. [56]

    Audiolm: a language modeling approach to audio generation

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and ...

    2023

  49. [57]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04328, 2025

    2025 arXiv

  50. [58]

    Pvc: Progressive visual token compression for unified image and video processing in large vision-language models

    Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai. Pvc: Progressive visual token compression for unified image and video processing in large vision-language models. arXiv preprint arXiv:2412.09613, 2024

    2024 arXiv

  51. [59]

    Adaptive keyframe sampling for long video understanding

    Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. arXiv preprint arXiv:2502.21271, 2025

    2025 arXiv

  52. [60]

    Frag: Frame selection augmented generation for long video and long document understanding

    De-An Huang, Subhashree Radhakrishnan, Zhiding Yu, and Jan Kautz. Frag: Frame selection augmented generation for long video and long document understanding. arXiv preprint arXiv:2504.17447, 2025

    2025 arXiv

  53. [61]

    Re-thinking temporal search for long- form video understanding

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, et al. Re-thinking temporal search for long- form video understanding. In Proceedings of the Computer Vision and Patter...

    2025

  54. [62]

    Flexselect: Flexible token selection for efficient long video understanding

    Yunzhu Zhang, Yu Lu, Tianyi Wang, Fengyun Rao, Yi Yang, and Linchao Zhu. Flexselect: Flexible token selection for efficient long video understanding. arXiv preprint arXiv:2506.00993, 2025

    2025 arXiv

  55. [63]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  56. [64]

    Video-r1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  57. [65]

    Llava-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 2024. 15

    2024 arXiv

  58. [66]

    Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning

    Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou. Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning. arXiv preprint arXiv:2505.12434, 2025

    2025

  59. [67]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  60. [68]

    Herm: Benchmarking and enhancing multimodal llms for human-centric understanding

    Keliang Li, Zaifei Yang, Jiahe Zhao, Hongze Shen, Ruibing Hou, Hong Chang, Shiguang Shan, and Xilin Chen. Herm: Benchmarking and enhancing multimodal llms for human-centric understanding. arXiv preprint arXiv:2410.06777, 2024

    2024 arXiv

  61. [69]

    Mc-llava: Multi-concept personalized vision-language model

    Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11706, 2024

    2024

  62. [70]

    Probio: A protocol-guided multimodal dataset for molecular biology lab

    Jieming Cui, Ziren Gong, Baoxiong Jia, Siyuan Huang, Zilong Zheng, Jianzhu Ma, and Yixin Zhu. Probio: A protocol-guided multimodal dataset for molecular biology lab. Advances in Neural Information Processing Systems, 36:41543–41571, 2023

    2023

  63. [71]

    Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos

    Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24129–24138, 2025

    2025

  64. [72]

    Chathuman: Chatting about 3d humans with tools

    Jing Lin, Yao Feng, Weiyang Liu, and Michael J Black. Chathuman: Chatting about 3d humans with tools. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8150–8161, 2025

    2025

  65. [73]

    Unipose: A unified multimodal framework for human pose comprehension, generation and editing

    Yiheng Li, Ruibing Hou, Hong Chang, Shiguang Shan, and Xilin Chen. Unipose: A unified multimodal framework for human pose comprehension, generation and editing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27805–27815, 2025

    2025

  66. [74]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067–20079, 2023

    2023

  67. [75]

    Motionllm: Understanding human behaviors from human motions and videos

    Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.20340, 2024

    2024 arXiv

  68. [76]

    Egolm: Multi-modal language model of egocentric motions

    Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions. arXiv preprint arXiv:2409.18127, 2024

    2024 arXiv

  69. [77]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  70. [78]

    Mmcode: Benchmarking multimodal large language models for code generation with visually rich programming problems

    Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, and Jing Ma. Mmcode: Benchmarking multimodal large language models for code generation with visually rich programming problems. arXiv preprint arXiv:2404.09486, 2024

    2024 arXiv

  71. [79]

    Mmiu: Multimodal multi-image understanding for evaluating large vision- language models

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision- language models. arXiv preprint arXiv:2408.02718, 2024

    2024 arXiv

  72. [80]

    Hourvideo: 1-hour video-language understanding

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video-language understanding. Advances in Neural Information Processing Systems, 37:53168–53197, 2024

    2024

  73. [81]

    Video simpleqa: Towards factuality evaluation in large video language models

    Meng Cao, Pengfei Hu, Yingyao Wang, Jihao Gu, Haoran Tang, Haoze Zhao, Jiahua Dong, Wangbo Yu, Ge Zhang, Ian Reid, et al. Video simpleqa: Towards factuality evaluation in large video language models. arXiv preprint arXiv:2503.18923, 2025

    2025 arXiv

  74. [82]

    Video-mmlu: A massive multi-discipline lecture understanding benchmark

    Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video-mmlu: A massive multi-discipline lecture understanding benchmark. arXiv preprint arXiv:2504.14693, 2025

    2025 arXiv

  75. [83]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098–89124, 2024

    2024

  76. [84]

    Grounded multi-hop videoqa in long-form egocentric videos

    Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi-hop videoqa in long-form egocentric videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2159–2167, 2025. 16

    2025

  77. [85]

    Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos

    Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arXiv preprint arXiv:2406.08407, 2024

    2024 arXiv

  78. [86]

    Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. arXiv preprint arXiv:2411.14794, 2024

    2024 arXiv

  79. [87]

    Videoreasonbench: Can mllms perform vision-centric complex video reasoning? arXiv preprint arXiv:2505.23359, 2025

    Yuanxin Liu, Kun Ouyang, Haoning Wu, Yi Liu, Lin Sui, Xinhao Li, Yan Zhong, Y Charles, Xinyu Zhou, and Xu Sun. Videoreasonbench: Can mllms perform vision-centric complex video reasoning? arXiv preprint arXiv:2505.23359, 2025

    2025

  80. [88]

    Aria: An open multimodal native mixture-of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, et al. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024

    2024 arXiv

  81. [89]

    Longvila: Scaling long-context visual language models for long videos

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024

    2024 arXiv

  82. [90]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024

    2024 arXiv

  83. [91]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  84. [92]

    Minicpm-o: An open-source tool for compressing llms with orthogonal projection

    OpenBMB. Minicpm-o: An open-source tool for compressing llms with orthogonal projection. https: //github.com/OpenBMB/MiniCPM-o, 2024. Accessed: 2024-05-21

    2024

  85. [93]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024

    2024 arXiv

  86. [94]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  87. [95]

    ByteDance. Doubao. https://www.volcengine.com/product/doubao, 2024

    2024

  88. [96]

    Grok-2 beta release

    xAI. Grok-2 beta release. https://x.ai/news/grok-2, May 2024

    2024

  89. [97]

    Introducing Claude 3.5 Sonnet

    Anthropic. Introducing Claude 3.5 Sonnet. https://www.anthropic.com/news/ 3-5-models-and-computer-use , June 2024

    2024

  90. [98]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  91. [99]

    Introducing gemini 2.0: Our new ai model for the agentic era

    Google DeepMind. Introducing gemini 2.0: Our new ai model for the agentic era. https://blog. google/technology/google-deepmind/google-gemini-ai-update-december-2024/ , 2024

    2024

  92. [100]

    Qvq-max: Think with evidence

    QwenLM. Qvq-max: Think with evidence. https://qwenlm.github.io/blog/qvq-max-preview/ , 2025

    2025

  93. [101]

    Claude 3.7 sonnet and claude code

    Anthropic. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/ claude-3-7-sonnet , February 2025

    2025

  94. [102]

    Introducing OpenAI o3-mini

    OpenAI. Introducing OpenAI o3-mini. https://openai.com/index/openai-o3-mini/ , 2025

    2025

  95. [103]

    Internlm-xcomposer2

    Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. Internlm-xcomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions. arXiv preprint arXiv:2412.09596, 2024

    2024 arXiv

  96. [104]

    Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction

    Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. arXiv preprint arXiv:2501.03218, 2025. 17

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.