Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims MLLMs score 61.83% on egocentric object cognition vs. humans' 94.63%, with time perception the weakest skill.

desk verdict A genuinely useful new egocentric object-cognition benchmark, but the evaluation protocol confounds model-family comparisons and the Past-dimension visual prompts need validation before the headline numbers are taken at face value. read the letter →

arxiv 2506.05287 v1 pith:ZE4BQAXO submitted 2025-06-05 cs.CV

Yuqian Yuan , Ronghao Dang , Long Li , Wentong Li , Dian Jiao , Xin Li , Deli Zhao , Fan Wang
show 3 more authors
Wenqiao Zhang Jun Xiao Yueting Zhuang
This is my paper
classification cs.CV
keywords EOC-Benchegocentricvisionvideoquestionansweringmultimodallargelanguagemodelsobject-centriccognitiontemporalperceptionbenchmarkevaluationvisualprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EOC-Bench is a new benchmark for testing whether multimodal large language models (MLLMs) can identify, recall, and forecast objects in first-person, egocentric videos where objects are moved, handled, and occluded. The paper claims that current MLLMs lag human performance by a wide margin: the best proprietary model, GPT-4o, scores 61.83% on 3,277 question-answer pairs while human annotators reach 94.63%. The largest shortfall is Absolute Time Perception, where the best model reaches only 34.46%, suggesting that MLLMs cannot reliably answer 'how long ago' or 'how many seconds' type questions. If the benchmark is accepted as valid, it establishes that dynamic object cognition over time is a missing capability in current embodied AI systems.

What carries the argument

The evaluation apparatus has three load-bearing parts. First, a hierarchical taxonomy splits object cognition into Past, Present, and Future, with 11 task definitions that operationalize skills like retrospection and prediction. Second, visual object prompts—point, box, or mask overlaid on the last frame using set-of-mark prompting—replace text descriptions of objects, so that 'the leftmost bowl' remains unambiguous even after objects are rearranged. Third, the Multi-Scale Temporal Accuracy (MSTA) metric scores open-ended time questions by checking whether predicted timestamps fall within relative error margins of 1%, 10%, 20%, and 30% of the ground-truth duration, averaging across the four scales. Together these convert continuous, cluttered first-person interactions into scorable probes.

What would settle it

Re-answer a sample of Past questions with fresh annotators who watch the full video and must first localize the prompted object in every frame where it appears; if their answers disagree with the released labels on more than a small fraction (say, 10%) of the sampled questions, the Past dimension's ground truth does not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that object cognition in egocentric scenes must be evaluated across three temporal categories—Past, Present, and Future—and that doing so reveals a systematic weakness in MLLMs. EOC-Bench contains 3,277 QA pairs across 656 real-world videos, spanning 11 fine-grained dimensions such as Object State Retrospection, Anomaly Perception, and Trajectory and Motion Prediction, with each question anchored by a visual point, box, or mask prompt on the final frame. Across the evaluated models, the best proprietary model (GPT-4o) scores 61.83%, the best open-source model (InternVL2.5-78B) scores 52.33%, and humans score 94.63%. The paper interprets the large gap, especially the 34.46% best score on Absolute Time Perception, as evidence that MLLMs lack persistent object tracking and temporal memory, which are prerequisites for embodied assistants and robots.

Load-bearing premise

The benchmark assumes that a marker drawn on the final frame of a video unambiguously identifies the same object throughout the entire video, even when that object moves, is occluded, or changes appearance in earlier frames.

Editorial extensions

If this is right

  • If the benchmark is valid, embodied assistants cannot rely on current MLLMs for object tracking through manipulation; past-oriented memory is the primary bottleneck.
  • Absolute Time Perception at 34.46% best model score means no evaluated model can reliably report how long ago an event happened, a skill required for long-horizon tasks.
  • Multi-frame inputs help: GPT-4o and Gemini-2.0-flash gain 24.6 and 20.1 points from one frame to 32 frames, with Past-task gains of 49.2 and 60.2 points, so frame sampling is a major source of error.
  • Object-level video MLLMs such as VideoRefer outperform several general video models, indicating object-centric training helps, but they still fail on relationship evolution and absolute time.
  • Future-task performance improves with model scale across Qwen2.5-VL and VideoLLaMA3, while Past-task performance does not scale the same way, pointing to a structural memory limitation rather than a simple capacity issue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A re-annotation study that tracks each prompted object from the full video could reveal how often the final-frame prompt identifies the wrong object in earlier frames, which would determine the Past dimension's reliability.
  • Reporting per-video-source scores (EPIC-KITCHENS, Ego4D, Charades-ego, MECCANO, self-recorded) would show whether the reported gap is driven by specific scene types, such as industrial or self-recorded footage.
  • The MSTA thresholds are tuned to the error distribution of a few human annotators; if the distribution changes across tasks or languages, the 1%, 10%, 20%, 30% bands may need recalibration to keep scores comparable.
  • EOC-Bench could be turned into a training objective for embodied agents whose own actions alter the scene, converting the evaluation into an interactive benchmark that measures continual object understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. EOC-Bench is a new benchmark for object-centric question answering in dynamic egocentric videos, containing 3,277 human-annotated QA pairs over 656 videos from EPIC-KITCHENS, Ego4D, Charades-ego, MECCANO, and self-recorded clips. Questions are organized into Past, Present, and Future temporal categories covering 11 cognitive dimensions; objects are referenced by visual prompts (point, box, mask) overlaid on the final frame. The paper introduces a mixed-format annotation framework (true/false, single/multiple choice, open-ended) and a Multi-Scale Temporal Accuracy (MSTA) metric for open-ended time questions. The main empirical finding is that the best proprietary model (GPT-4o, 61.83%) substantially lags human performance (94.63%), with Absolute Time Perception being the hardest skill (best model 34.46%).

Significance. If the instrument is valid, EOC-Bench fills a genuine gap: existing embodied benchmarks mostly probe static scenes, whereas this benchmark targets persistent object understanding across dynamic egocentric interactions. The annotation pipeline is carefully described with concrete cross-checking statistics (185 object prompts reannotated, 34 discarded, 76 QA pairs collaboratively resolved), a human baseline is reported, and Table 7 gives reproducible model configurations. The release of code, benchmark, and project page is also a strength. However, the central validity question is whether final-frame-only visual prompts can reliably refer to objects queried about in earlier frames; this is not validated and directly threatens the Past dimension. The proprietary-versus-open-source comparison is additionally confounded by per-frame timestamp injection for proprietary models. These issues are substantial but fixable with additional validation and controlled ablations.

major comments (3)
  1. [§3.3, §C.3] The Past dimension, which contains 1,422 of the 3,277 QA pairs, relies on visual object prompts that are annotated only on the final frame of each video, as stated in §3.3 and operationalized in the system prompt of §C.3. Many Past questions, however, ask about an object's earlier state, location, or relationship (e.g., 'Where was <object 0> originally?' in Fig. 12 and 'What was the relationship between <object 0> and <object 1> before?' in Fig. 13). The benchmark therefore assumes that a single final-frame marker uniquely and persistently identifies the same physical object at the earlier time being queried. The quality-control pipeline in §3.2.3 and §A.2.1 checks prompt placement, format, and content, but it does not validate cross-frame identity consistency, nor does it measure how often the marked object is occluded, moved, or confused with an identical-looking neighbor before the final frame. This is not a hypothetical concern because the paper itself identifies visual ambiguity from similar objects in close proximity as a core egocentric challenge (§1). Without per-question tracking validation, Past scores may reflect tracking ambiguity rather than the intended cognitive ability, so the central claim about MLLMs' weakness in recalling past object states is not yet cleanly supported. I recommend adding a tracking-identity validation step (e.g., annotator confirmation that the marked object is traceable to the queried moment, or per-frame re-identification), reporting the rate of occlusion/swap cases, and either excluding or re-annotating ambiguous questions.
  2. [§4.1, §C.2, Table 2] The evaluation protocol is asymmetric across model families. For proprietary models (GPT-4o, GPT-4o-mini, Gemini-2.0-flash) the paper states that 'we manually added a timestamp before each frame' (§C.2), and these entries are marked with an asterisk in Tables 2 and 3. Open-source models are evaluated without such timestamps. Because Absolute Time Perception and several Past tasks explicitly require temporal localization, providing frame timestamps to proprietary models is a substantial advantage, and the reported gap between proprietary and open-source models (e.g., GPT-4o at 61.83% vs. InternVL2.5-78B at 52.33%) is confounded by this difference. The paper should run proprietary models without timestamps and open-source models with the same timestamp injection, or otherwise restrict the comparison to equivalent input conditions. Without this control, the headline conclusion about proprietary superiority is not fully justified.
  3. [§3.2.4, §A.1] The MSTA metric in Eq. (1) uses relative error thresholds C = {1%, 10%, 20%, 30%}, and §A.1 explains that these thresholds are derived from a quantile analysis of error ratios from only three human volunteers. The same three volunteers provide the human performance reported in Table 2 (94.63%), so the human score is computed under a metric calibrated to the very same individuals' error distribution. This is a mild circularity and, more importantly, the choice of thresholds is not robustly justified: rounded quantiles from n=3 annotators are not a principled tolerance scheme. I request an ablation of human and model scores under alternative threshold sets (e.g., continuous interval scoring or 5%/15%/25%/35%), and a statement of how sensitive the conclusions are to the exact thresholds. At minimum, the paper should clarify that MSTA is a definitional evaluation metric, not a finding derived from the human-error analysis.
minor comments (6)
  1. [Table 3] The column header 'Multi-Choice Anwer' contains a typo and should read 'Multi-Choice Answer'.
  2. [Figure 2] The percentages in Figure 2 (34%, 28%, 24%, 13%, 1%) are not explained in the caption or text; please state explicitly which variable these percentages refer to (e.g., video source distribution or scenario categories) and add a legend or explicit mapping.
  3. [§3.2.3] The paper reports cross-checking and collaborative resolution counts but does not report inter-annotator agreement (e.g., Cohen's kappa or percentage agreement) for question-answer labels; adding this would strengthen the quality-control section.
  4. [Table 2 caption and §4.2] The caption says grey marks denote results below random guess and also says entries in grey indicate image-level methods using only the last frame; these two definitions are contradictory (e.g., Osprey-7B has a mean above random but is grey). Please disambiguate the grey-mark semantics.
  5. [Appendix A.1, Appendix D] There are minor typos: 'threhold' should be 'threshold' in A.1, and 'worldcloud' should be 'word cloud' in D. The sentence in Section 1 ('significantly lags behind human-level performance') should agree in number ('models ... lag').
  6. [§4.5, Table 4] The 'γ' column is labeled as 'rate of increase' but the values are percentage-point differences (e.g., 61.8 - 49.6 = 12.2, not a percentage increase of 24.6%); please clarify the computation or rename the column.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity: MSTA thresholds are fitted to volunteer error ratios, and the same volunteer performance is then scored with those thresholds; the core benchmark and MLLM-vs-human gap remain largely independent.

  1. fitted input called prediction [Section 3.2.4 (Eq. 1), Section A.1, Section 4.1 (Baselines)]
    "These thresholds are derived from human error analysis... we first asked three volunteers to answer this type of question, and then analyzed the error ratio compared to the ground truth... Based on these analyses, we set the threshold for dynamic error margins at {1%, 10%, 20%, 30%}... a 30% threshold ... accommodating almost all human answers within this margin. ... we also assess human performance on EOC-Bench using video input with three volunteers."

    The Multi-Scale Temporal Accuracy metric (Eq. 1) uses thresholds {1%,10%,20%,30%} that are selected as quantiles (50%,75%,90%,95%) of the error ratios of three volunteers answering the Open-Ended temporal questions. Human performance is then measured on the same benchmark with the same volunteer-based protocol and scored with the same calibrated thresholds. Thus the human OQ/ATP reference score is not an independent yardstick: the thresholds were explicitly chosen to accommodate almost all human answers, so the reported human score is partly constructed from the same human data used to define the metric.

full rationale

EOC-Bench is primarily a new measurement instrument rather than a derivation from assumptions, so most of the paper is self-contained. The benchmark is evaluated against external MLLM baselines and compared to human performance; no central result is imported from the authors' prior work. Self-citations to ECBench, VideoRefer, VideoLLaMA, and Osprey are used as related work or as evaluated baselines, not as load-bearing justification for the benchmark's validity. The visual prompts being annotated only on the final frame is a validity risk for Past-oriented questions, but it is not a circularity in the derivation-to-input sense—there is no equation or fitted parameter that makes the Past scores equivalent to the final-frame annotation by construction. The one identifiable circular step is the calibration of MSTA thresholds from volunteer error ratios and the subsequent use of the same volunteer protocol as the human reference score. Because this only affects the open-ended temporal subset and does not force the model scores, the circularity is minor; the overall benchmark conclusion remains substantially evidence-based.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of the annotation process and the fairness of the evaluation protocol. The only explicit numeric parameter fitted to data is the MSTA threshold set, chosen from three volunteers' error quantiles. The benchmark assumes human annotations are ground truth, that final-frame visual prompts refer reliably to the same object over time, and that the 11-task taxonomy covers the abilities of interest. No new physical entities are introduced; EOC-Bench and MSTA are measurement resources, not invented entities.

free parameters (1)
  • MSTA relative error tolerance thresholds = {1%, 10%, 20%, 30%}
    Selected from quantile analysis of error ratios from three human volunteers (50%, 75%, 90%, 95% quantiles gave 1.4%, 11.9%, 20%, 30%, rounded). The metric's tolerance boundaries depend on this hand-chosen set.
assumptions (4)
  • domain assumption Human annotations are treated as ground truth for all 3,277 QA pairs.
    The benchmark's validity rests on annotator judgments being correct. The paper mitigates this with cross-checking and collaborative resolution of 76 questions (Section A.2.1).
  • domain assumption The 11-dimension Past/Present/Future taxonomy covers the relevant object-cognition abilities for embodied systems.
    The taxonomy is proposed in Section 3.2.2 and inspired by MVBench and MMbench-Video; no external criterion guarantees completeness.
  • domain assumption Visual prompts overlaid on the final frame identify the same object across the whole video.
    Section 3.3 states prompts are annotated on the final frame; questions about past states assume persistent reference.
  • domain assumption Zero-shot evaluation with default model settings is representative of a model's object-cognition ability.
    All models are evaluated zero-shot (Section 4.1); results may change with prompting or fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?." pith.science (2026). https://pith.science/paper/ZE4BQAXO

@misc{pith2026250605287,
  author       = {Pith},
  title        = {Pith review of: EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZE4BQAXO}},
  note         = {Machine review of arXiv:2506.05287}
}
read the original abstract

The emergence of multimodal large language models (MLLMs) has driven breakthroughs in egocentric vision applications. These applications necessitate persistent, context-aware understanding of objects, as users interact with tools in dynamic and cluttered environments. However, existing embodied benchmarks primarily focus on static scene exploration, emphasizing object's appearance and spatial attributes while neglecting the assessment of dynamic changes arising from users' interactions. To address this gap, we introduce EOC-Bench, an innovative benchmark designed to systematically evaluate object-centric embodied cognition in dynamic egocentric scenarios. Specially, EOC-Bench features 3,277 meticulously annotated QA pairs categorized into three temporal categories: Past, Present, and Future, covering 11 fine-grained evaluation dimensions and 3 visual object referencing types. To ensure thorough assessment, we develop a mixed-format human-in-the-loop annotation framework with four types of questions and design a novel multi-scale temporal accuracy metric for open-ended temporal evaluation. Based on EOC-Bench, we conduct comprehensive evaluations of various proprietary, open-source, and object-level MLLMs. EOC-Bench serves as a crucial tool for advancing the embodied object cognitive capabilities of MLLMs, establishing a robust foundation for developing reliable core models for embodied systems.

Figures

Figures reproduced from arXiv: 2506.05287 by the authors.

Figure 1
Figure 1. Overview of EOC-Bench. EOC-Bench assesses Embodied Object Cognition capabilities of MLLMs in egocentric videos across three dimensions - Past, Present, and Future - encompassing 11 categories. EOC-Bench includes 3,277 object-level QA pairs utilizing a mixed-format human-in￾the-loop annotation framework across diverse tasks and conditions. EOC-Bench aims to reveal the limitations of MLLMs and promote the development … view at source ↗
Figure 2
Figure 2. Overall data distribution of EOC-Bench. (a) EOC-Bench encompasses three temporal dimensions: Past, Present, and Future, comprehensively evaluating 11 embodied cognitive abilities. (b) The dataset comprises videos from four distinct open video sources as well as self-recorded videos. (c) It spans a wide range of scenarios, offering a rich diversity of contexts for analysis. 2.2 Embodied Video Understanding Benchmarks… view at source ↗
Figure 3
Figure 3. Statistic analysis of EOC-Bench: (a) substantial diversity in object categories and usage taxonomies, (b) a wide range of video durations correlated with question count, and (c) a balanced distribution of response options across each question type. relies on the model’s ability to utilize physical laws and common sense knowledge for prediction and inference. This dimension is divided into three categories: • Traject… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Comparison of mainstream MLLMs on EOC-Bench. Left: Performance on 11 evaluation tasks within EOC-Bench. Right: Performance across different question types spanning Past, Present and Future categories. 4.4 Qualitative Results To intuitively showcase the performance of m…
Figure 5
Figure 5. Figure 5: Statistics distribution of human error ratio. (a) displays a histogram depicting the density and spread of human error ratios; (b) presents a pie chart categorizing the error ratios into quantitative segments. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Quantitative error analysis by type for choice-based questions in [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Quantitative error analysis for open-ended questions in [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Illustrative examples of annotation formats for visual prompts. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Worldcloud of questions and answers in EOC-Bench. • Object Location Retrospection ( [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Failure cases of the top-performing GPT-4o on [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Visualization of samples in Object State Retrospection (Past). 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Visualization of samples in Location Retrospection (Past). 23 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Visualization of samples in Object Relationship Evolution (Past). 24 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Visualization of samples in Absolute Time Perception (Past). 25 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Visualization of samples in Immediate State Recognition (Present). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Visualization of samples in Object Relationship (Present). 27 [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: Visualization of samples in Purpose and Function Inference (Present). 28 [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 18
Figure 18. Figure 18: Visualization of samples in Anomaly Perception (Present). 29 [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: Visualization of samples in Trajectory and Motion Prediction (Future). 30 [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 20
Figure 20. Figure 20: Visualization of samples in State Change Prediction (Future). 31 [PITH_FULL_IMAGE:figures/full_fig_p031_20.png]
Figure 21
Figure 21. Figure 21: Visualization of samples in Dynamic Relationship Prediction (Future). 32 [PITH_FULL_IMAGE:figures/full_fig_p032_21.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Graft merges two domain-specialized multimodal models by combining channel-wise gating, entropy-based global weighting, and an activation compatibility score to improve fusion without retraining.

Reference graph

Works this paper leans on

65 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  3. [3]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  4. [4]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  5. [5]

    Aria digital twin: A new benchmark dataset for egocentric 3d machine perception

    Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 20133–20143, 2023

    2023

  6. [6]

    Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments

    Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. InConference on robot learning, pages 477–490. PMLR, 2022

    2022

  7. [7]

    Vinci: A real-time embodied smart assistant based on egocentric vision-language model.arXiv preprint arXiv:2412.21080, 2024

    Yifei Huang, Jilan Xu, Baoqi Pei, Yuping He, Guo Chen, Lijin Yang, Xinyuan Chen, Yaohui Wang, Zheng Nie, Jinyao Liu, et al. Vinci: A real-time embodied smart assistant based on egocentric vision-language model.arXiv preprint arXiv:2412.21080, 2024

    arXiv 2024

  8. [8]

    A survey of embodied learning for object-centric robotic manipulation.arXiv preprint arXiv:2408.11537, 2024

    Ying Zheng, Lei Yao, Yuejiao Su, Yi Zhang, Yi Wang, Sicheng Zhao, Yiyi Zhang, and Lap-Pui Chau. A survey of embodied learning for object-centric robotic manipulation.arXiv preprint arXiv:2408.11537, 2024

    arXiv 2024

Show all 65 references
  1. [9]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation

    Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. InCVPR, pages 18061–18070, 2024. 10

    2024

  2. [10]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. InICML, pages 8469–8488. PMLR, 2023

    2023

  3. [11]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InCVPR, 2024

    2024

  4. [12]

    Videorefer suite: Advancing spatial-temporal object understanding with video llm

    Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video llm. CVPR, 2025

    2025

  5. [13]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. InICCV, pages 5267–5275, 2017

    2017

  6. [14]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. InCVPR, pages 19129–19139, 2022

    2022

  7. [15]

    Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    2022 arXiv

  8. [16]

    Robovqa: Multimodal long-horizon reasoning for robotics

    Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. InICRA, pages 645–652, 2024

    2024

  9. [17]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, et al. Openeqa: Embodied question answering in the era of foundation models. InCVPR, 2024

    2024

  10. [18]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. InCVPR, 2025

    2025

  11. [19]

    Ecbench: Can multi-modal foundation models understand the egocentric world? a holistic embodied cognition benchmark

    Ronghao Dang, Yuqian Yuan, Wenqi Zhang, Yifei Xin, Boqiang Zhang, Long Li, et al. Ecbench: Can multi-modal foundation models understand the egocentric world? a holistic embodied cognition benchmark. InCVPR, 2025

    2025

  12. [20]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  13. [21]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    2025 arXiv

  14. [22]

    Osprey: Pixel understanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. InCVPR, pages 28202–28211, 2024

    2024

  15. [23]

    Vip-llava: Making large multimodal models understand arbitrary visual prompts

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts. InCVPR, pages 12914–12923, 2024

    2024

  16. [24]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  17. [25]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  18. [26]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  19. [27]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  20. [28]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  21. [29]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023. 11

    2023 arXiv

  22. [30]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  23. [31]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, et al. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  24. [32]

    Tokenpacker: Efficient visual projector for multimodal llm.arXiv preprint arXiv:2407.02392, 2024

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm.arXiv preprint arXiv:2407.02392, 2024

    2024 arXiv

  25. [33]

    Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation

    Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, Siliang Tang, Jun Xiao, Hui Lin, Yueting Zhuang, and Beng Chin Ooi. Healthgpt: A medical large vision-language model for unifying comprehension and generat...

    2025

  26. [34]

    Hyperllava: Dynamic visual and language expert tuning for multimodal large language models, 2024

    Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, Juncheng Li, Siliang Tang, and Yueting Zhuang. Hyperllava: Dynamic visual and language expert tuning for multimodal large language models, 2024

    2024

  27. [35]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  28. [36]

    Mmbench- video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench- video: A long-form multi-shot benchmark for holistic video understanding. InNeurIPS, 2025. [37]Perception test: A diagnostic benchmark for multimodal video models, 2023

    2025

  29. [38]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. InCVPR, pages 961–970, 2015

    2015

  30. [39]

    A benchmark dataset and comparison study for multi-modal human action analytics.TOMM, 2020

    Jiaying Liu, Sijie Song, Chunhui Liu, Yanghao Li, and Yueyu Hu. A benchmark dataset and comparison study for multi-modal human action analytics.TOMM, 2020

    2020

  31. [40]

    A large-scale study of spatiotemporal representation learning with a new benchmark on action recognition

    Andong Deng, Taojiannan Yang, and Chen Chen. A large-scale study of spatiotemporal representation learning with a new benchmark on action recognition. InICCV, pages 20519–20531, 2023

    2023

  32. [41]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. InICCV, pages 706–715, 2017

    2017

  33. [42]

    Abstractive multi-video captioning: Benchmark dataset construction and extensive evaluation

    Rikito Takahashi, Hirokazu Kiyomaru, Chenhui Chu, and Sadao Kurohashi. Abstractive multi-video captioning: Benchmark dataset construction and extensive evaluation. InLREC-COLING, pages 57–69, 2024

    2024

  34. [43]

    Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024

    2024 arXiv

  35. [44]

    Vilma: A zero-shot benchmark for linguistic and temporal grounding in video-language models.arXiv preprint arXiv:2311.07022, 2023

    Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, et al. Vilma: A zero-shot benchmark for linguistic and temporal grounding in video-language models.arXiv preprint arXiv:2311.07022, 2023

    2023 arXiv

  36. [45]

    Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  37. [46]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. InCVPR, pages 9777–9786, 2021

    2021

  38. [47]

    Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models

    Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. In ICLR, 2025

    2025

  39. [48]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, et al. Mlvu: A comprehensive benchmark for multi-task long video understanding. InCVPR, 2025. [49]Longvideobench: A benchmark for long-context interleaved video-language understanding, 2025

    2025

  40. [50]

    Movqa: A benchmark of versatile question-answering for long-form movie understanding.arXiv preprint arXiv:2312.04817, 2023

    Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, et al. Movqa: A benchmark of versatile question-answering for long-form movie understanding.arXiv preprint arXiv:2312.04817, 2023. 12

    2023 arXiv

  41. [51]

    Infinibench: A comprehensive benchmark for large multimodal models in very long video understanding

    Kirolos Ataallah, Chenhui Gou, Eslam Abdelrahman, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. Infinibench: A comprehensive benchmark for large multimodal models in very long video understanding. arXiv preprint arXiv:2406.19875, 2024

    2024

  42. [52]

    Mmvu: Measuring expert-level multi-discipline video understanding.arXiv preprint arXiv:2501.12380, 2025

    Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, et al. Mmvu: Measuring expert-level multi-discipline video understanding.arXiv preprint arXiv:2501.12380, 2025

    2025 arXiv

  43. [53]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.arXiv preprint arXiv:2501.13826, 2025

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  44. [54]

    Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments

    Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments. InICCV, pages 1675–1685, 2021

    2021

  45. [55]

    Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models. CoRR, 2023

    2023

  46. [56]

    Egoplan-bench2: A benchmark for multimodal large language model planning in real-world scenarios.arXiv preprint arXiv:2412.04447, 2024

    Lu Qiu, Yi Chen, Yuying Ge, Yixiao Ge, Ying Shan, and Xihui Liu. Egoplan-bench2: A benchmark for multimodal large language model planning in real-world scenarios.arXiv preprint arXiv:2412.04447, 2024

    2024 arXiv

  47. [57]

    Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain.arXiv preprint arXiv:2402.15527, 2024

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain.arXiv preprint arXiv:2402.15527, 2024

    2024 arXiv

  48. [58]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. InNeurIPS, 2023

    2023

  49. [59]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InCVPR, pages 18995–19012, 2022

    2022

  50. [60]

    Sti-bench: Are mllms ready for precise spatial-temporal world understanding?arXiv preprint arXiv:2503.23765, 2025

    Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu, Wenxiao Cai, Zheng Liu, and Bo Zhao. Sti-bench: Are mllms ready for precise spatial-temporal world understanding?arXiv preprint arXiv:2503.23765, 2025

    2025 arXiv

  51. [61]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. InECCV, pages 720–736, 2018

    2018

  52. [62]

    Charades-ego: A large-scale dataset of paired third and first person videos.arXiv preprint arXiv:1804.09626, 2018

    Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos.arXiv preprint arXiv:1804.09626, 2018

    2018 arXiv

  53. [63]

    The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain

    Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In WACV, 2021

    2021

  54. [64]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  55. [65]

    Nvila: Efficient frontier visual language models.arXiv preprint arXiv:2412.04468, 2024

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, et al. Nvila: Efficient frontier visual language models.arXiv preprint arXiv:2412.04468, 2024

    2024 arXiv

  56. [66]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want. InICLR, 2025

    2025

  57. [67]

    Do Sample

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023. 13 Appendix In this document, we offer additional details about our benchmark. The Appen...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.