Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces All-Angles Bench, a 2,132-question multi-view benchmark, and shows that leading multimodal models—InternVL2.5-38B, Qwen2.5-VL-72B, Gemini-2.0-Flash—score 55–61 on a human-evaluated subset where humans average 82, with…

desk verdict A genuinely useful multi-view benchmark with a compelling main result, but the human-baseline statistics are too shaky to pin down the exact gap as reported. read the letter →

arxiv 2504.15280 v2 pith:WSX4VCGE submitted 2025-04-21 cs.CV cs.CL

Chun-Hsiao Yeh , Chenyu Wang , Shengbang Tong , Ta-Ying Cheng , Ruoyu Wang , Tianzhe Chu , Yuexiang Zhai , Yubei Chen
show 2 more authors
Shenghua Gao Yi Ma
This is my paper · ORCID
classification cs.CVcs.CL
keywords multi-viewunderstandingmultimodallargelanguagemodelsbenchmarkcameraposeestimationcross-viewcorrespondencegeometricreasoningpairedquestionsevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Current multimodal large language models (MLLMs) do not understand a 3D scene from multiple camera views the way people do, and the paper's aim is to make that gap measurable and precise. It builds All-Angles Bench, a benchmark of 2,132 human-annotated multiple-choice questions over 90 real-world scenes, covering counting, attribute identification, relative distance, relative direction, object manipulation, and camera pose estimation. On a 250-question subset that humans answered at 82.0% accuracy, the best tested models—InternVL2.5-38B, Ovis2-16B, Qwen2.5-VL-72B, and Gemini-2.0-Flash—stay between roughly 55% and 61%, and several models score below chance on camera pose estimation. The paper argues that the two specific weak points are cross-view correspondence when objects are partially occluded and coarse camera pose estimation, and that chain-of-thought prompting does not reliably close the gap.

What carries the argument

The load-bearing object is All-Angles Bench itself: 90 curated real-world scenes with 2,132 three-option multiple-choice questions, each refined and verified by eight PhD annotators. Its central design device is paired-question generation—each of 85.3% of questions has a twin that rephrases the wording or swaps the referenced views while preserving the same visual correspondence. Comparing success on the two versions yields an 'inconsistent' (IC) score, which exposes whether a model's correct answer reflects robust multi-view understanding or a brittle inference. A second analytical instrument is a visualization prompt that asks the model to lay out objects and camera positions on a 10×10 grid; this reveals systematic misalignment in camera coordinates and viewpoint transformation.

What would settle it

Run a fresh panel of at least five independent annotators on a randomly selected 250-question subset, computing inter-annotator agreement; if their average accuracy falls more than 10 points short of the paper's 82.0 human baseline, or if any MLLM reaches within 5 points of the new baseline on a held-out set, the central gap claim would be overturned.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that current MLLMs remain far from human-level proficiency in multi-view understanding. On the 250-question human-evaluated subset, the human baseline is 82.0 average accuracy while the best model, InternVL2.5-38B, reaches 60.8, a gap of more than 20 points. The largest single failure is camera pose estimation: humans score 88.9, but the best MLLMs reach only the 30–50 range and many open-source models are at or below the 33.3% random-guess level. The paper also shows via a paired-question scheme that MLLMs often answer semantically equivalent rephrased questions inconsistently, meaning a correct answer is frequently a lucky guess rather than evidence of genuine multi-view understanding. The conclusion is that domain-specific refinements or modules that embed stronger multi-view awareness are necessary.

Load-bearing premise

For the measured 20-point gap to mean what the paper says, the human-written ground-truth answers must be correct on nearly every question, and the two-person human baseline of 82.0 must be a faithful estimate of human-level performance; if either fails, the gap and the conclusion could be inflated.

Editorial extensions

If this is right

  • All-Angles Bench gives the community a shared yardstick for multi-view reasoning, so claims about embodied or spatial competence in MLLMs can be checked against a fixed, human-verified question set.
  • Because camera pose estimation is where the biggest gap appears, camera-pose ordering can serve as a fast diagnostic test for multi-view geometric awareness in future model releases.
  • The paired-question inconsistency score shows that single-question accuracy overstates multi-view understanding; reporting both accuracy and consistency becomes the norm for such benchmarks.
  • The failure of chain-of-thought prompting to consistently help implies that the bottleneck is perceptual alignment across views rather than language-level reasoning, steering future work toward domain-specific training or geometric modules.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate editorial test: recruit a larger panel of independent annotators (say 20 or more) for a fresh subset and measure inter-annotator agreement; if agreement is low on the hardest items, the 82.0 human baseline may be optimistic and the 20-point gap somewhat shrinks.
  • The paired-question design could be extended into a graded consistency metric—measuring how many view permutations and phrasing variants a model survives—which might predict reliability as an embodied agent better than raw accuracy.
  • A direct test of the paper's proposed remedy: fine-tune one of the evaluated open-source families on synthetic multi-view data with explicit 3D ground truth and see whether its All-Angles score jumps past the current 60.8 ceiling; this would separate the 'training data' explanation from the 'architecture' explanation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces All-Angles Bench, a benchmark of over 2,100 human-annotated multi-view question-answer pairs across 90 real-world scenes, spanning six tasks: counting, attribute identification, relative distance, relative direction, object manipulation, and camera pose estimation. The authors evaluate 27 closed- and open-source MLLMs with greedy decoding and an LLM-based answer extractor, and compare against a human baseline on a 250-question subset. They report a substantial human–model gap, identify cross-view correspondence and camera pose estimation as the main failure modes, and show that chain-of-thought prompting yields limited and model-dependent improvements. The central claim is that current MLLMs remain far from human-level multi-view understanding.

Significance. If the benchmark is reliable, it fills a clear gap: most existing evaluations of spatial reasoning in MLLMs are single-view or temporal, whereas All-Angles Bench directly tests geometric correspondence and cross-view consistency. The public release, the breadth of models evaluated, the paired-question consistency design, and the careful annotation effort (eight PhD annotators, human refinement of LLM-generated questions, cross-checking) are concrete strengths. The identification of camera pose estimation as a systemic weakness, with many models below random guessing, is a valuable and falsifiable finding for the embodied-AI community. However, the quantitative headline, particularly the 82.0% human vs. roughly 60% best-MLLM gap, rests on a 250-question human baseline whose exact protocol is contradictory in the manuscript and for which no agreement or uncertainty measures are reported.

major comments (3)
  1. [§3.1 vs. Supplement §8.1] The human evaluation protocol is described inconsistently. §3.1 states that human annotators 'each of whom independently answers every question,' while Supplement §8.1 states that each of the two evaluators was assigned 125 questions, i.e., disjoint halves with no overlap. If the latter is correct, the per-task human accuracies in Table 1 are computed on very small subsets and inter-annotator agreement cannot be assessed at all. This is load-bearing for the headline claim of a 82.0 vs. ~60.8 human–model gap. Please clarify the exact protocol, provide the raw per-question human answers, and if overlap exists, report a chance-corrected agreement statistic. If the disjoint design is kept, the human baseline should be re-estimated with confidence intervals and the text should be corrected.
  2. [Table 1 and Table 2 (250-question subset)] All human and model accuracies on the 250-question subset are reported as point estimates without confidence intervals or significance tests. Based on the task distribution in Figure 4, per-task sample sizes are roughly 20–60 questions; for example, camera pose estimation has about 18 questions in this subset. Consequently, a difference of a few percentage points between models, and even the ordering of some models, may reflect sampling noise. The human–model gap is large enough to survive modest noise for several tasks, but the precise magnitudes (e.g., 82.0% vs. 60.8% on average, or 88.9% vs. 16.7% on camera pose) are not statistically supported as reported. Please provide binomial confidence intervals or a significance test for the key comparisons, especially for the human baseline and the top-performing models.
  3. [Supplement §7.3 (annotation reliability)] The ground-truth answers for relative distance, relative direction, and object manipulation are annotator judgments, and the full 2,132-question set relies on eight PhD annotators with cross-checking but no reported inter-annotator agreement number. The supplement states that disagreements were resolved through group discussion, but the frequency and nature of disagreements are not quantified. Because the entire evaluation compares model accuracy against these subjective labels, an ambiguous-question analysis (e.g., Cohen's kappa on a shared subset, or annotation confidence scores) is required to rule out the possibility that annotation noise inflates the measured human–model gap. This is a correctness-risk concern, not an assertion that the labels are wrong; the gap is probably large enough to survive some noise, but the reported exact values need this support.
minor comments (5)
  1. [Supplement §7.3] The text reads 'we hiredeight Ph.D. students' — a typo for 'hired eight'.
  2. [Abstract and Table 1] The model name 'LLaV A-Onevision' appears in Table 1 and elsewhere; the standard spelling is 'LLaVA-OneVision', which should be used consistently.
  3. [Abstract] The phrase 'benchmark on27 representative MLLMs' is missing a space between 'on' and '27'.
  4. [Figure 1 and Section 2] The phrase 'top-bottom view' should be 'top-down view' in the camera pose estimation task description and figures.
  5. [Figure 5] The plot labeled 'Averaged IC' would be easier to interpret if the caption stated explicitly whether the average is a macro-average over the six tasks or a micro-average over questions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark annotations, not model outputs, define ground truth; the human-evaluation protocol discrepancy is a reproducibility concern, not a circular derivation.

full rationale

This is an empirical benchmark paper rather than a derivational one, so the circularity burden is low and is not met. The only potentially looping element is that GPT-4o generated initial questions and GPT-4o is later evaluated on the benchmark, but the paper breaks that loop by construction: every generated question was reviewed, rephrased, and re-annotated by eight PhD annotators, with the correct answer finalized by humans, not by the generating model. As the paper states, 'We hire eight human annotators who carefully examine each question along with the associated multi-view images, removing invalid entries and refining question phrasing... finally annotating the single correct answer.' The MLLM therefore supplies draft text, not ground truth, so the models' scores are measured against independent human judgments. The paired-question inconsistency metric is an internal consistency diagnostic and is not used as a fitted target or as the source of the headline claim. The human baseline is an empirical measurement on a 250-question subset, and the discrepancy between the main text ('each of whom independently answers every question') and the supplement ('Each evaluator was assigned 125 questions') is a legitimate reproducibility and reliability concern about inter-annotator agreement, but that is a correctness risk, not circularity. No free parameters are fitted, no prediction is derived from the benchmark's own outputs, and no load-bearing self-citation chain is present. The central claim that current MLLMs lag humans on this benchmark is therefore self-contained as an evaluation result.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is an empirical benchmark paper, not a derivation. The benchmark and metrics (IC inconsistency) are the contributed artifacts, and they depend on the two domain assumptions above. No free parameters are fitted to data, and no new physical or mathematical entities are introduced.

assumptions (2)
  • domain assumption Human-annotated answers are correct and unambiguous ground truth for multi-view understanding
    All evaluation scores compare model outputs against these annotations; annotation quality is verified by cross-checking but no inter-annotator agreement is reported (Supplement 7.3).
  • domain assumption The 250-question subset used for human comparison is representative of the full benchmark
    Human performance is measured on a randomly selected subset, and results are compared to MLLM performance on the same subset (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs." pith.science (2026). https://pith.science/paper/WSX4VCGE

@misc{pith2026250415280,
  author       = {Pith},
  title        = {Pith review of: Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSX4VCGE}},
  note         = {Machine review of arXiv:2504.15280}
}
read the original abstract

Multi-view understanding, the ability to reconcile visual information across diverse viewpoints for effective navigation, manipulation, and 3D scene comprehension, is a fundamental challenge in Multi-Modal Large Language Models (MLLMs) to be used as embodied agents. While recent MLLMs have shown impressive advances in high-level reasoning and planning, they frequently fall short when confronted with multi-view geometric consistency and cross-view correspondence. To comprehensively evaluate the challenges of MLLMs in multi-view scene reasoning, we propose All-Angles Bench, a benchmark of over 2,100 human carefully annotated multi-view question-answer pairs across 90 diverse real-world scenes. Our six tasks (counting, attribute identification, relative distance, relative direction, object manipulation, and camera pose estimation) specifically test model's geometric correspondence and the capacity to align information consistently across views. Our extensive experiments, benchmark on 27 representative MLLMs including Gemini-2.0-Flash, Claude-3.7-Sonnet, and GPT-4o against human evaluators reveals a substantial performance gap, indicating that current MLLMs remain far from human-level proficiency. Through in-depth analysis, we show that MLLMs are particularly underperforming under two aspects: (1) cross-view correspondence for partially occluded views and (2) establishing the coarse camera poses. These findings highlight the necessity of domain-specific refinements or modules that embed stronger multi-view awareness. We believe that our All-Angles Bench offers valuable insights and contribute to bridging the gap between MLLMs and human-level multi-view understanding. The project and benchmark are publicly available at https://danielchyeh.github.io/All-Angles-Bench/.

Figures

Figures reproduced from arXiv: 2504.15280 by the authors.

Figure 1
Figure 1. We present All-Angles Bench, a rich-annotated benchmark with over 2,100 Q&A pairs from 90 diverse scenes for evaluating multi-view understanding of MLLMs. Left and Middle: An example question setup of multiple views capturing the same scene and the corresponding questions. Right: Accuracies of six notable MLLMs across different question categories. Abstract Multi-view understanding, the ability to reconcile visual i… view at source ↗
Figure 2
Figure 2. Overview of All-Angles Bench. Our benchmark targets a comprehensive view of multi-view understanding, spanning six primary question types. These question types are designed to investigate several major aspects of 3D scene understanding, from creating correspondence between objects to associating relative object and camera poses. 1. Introduction Multi-view understanding is a fundamental challenge in bridging machine … view at source ↗
Figure 3
Figure 3. All-Angles Bench construction pipeline. (1) We collect and curate 90 diverse multi-view scenes and design six tasks that emphasize multi-view reasoning. (2) We generate initial questions via an MLLM, then refine and validate them through human annotation to ensure correctness, clarity, and domain relevance. (3) We create paired questions by systematically rephrasing or altering each view perspective while preserving… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Statistical overview of All-Angles Bench. The pie chart shows the distribution of 6 sub-tasks of multi-view understanding. The bar plot illustrates the percentage breakdown by primary and paired question-answers of each sub-task. vs. View 2). This process is similar to…
Figure 5
Figure 5. Figure 5: Paired question-answers inconsistency across 6 MLLMs. We report the proportions of IC and CC + WW. Notably, GPT-4o struggles with relative distance (around 70% inconsistency). Gemini-2.0-Flash and Claude-3.7-Sonnet exhibit more balanced performance, whereas Ovis2-34B a…
Figure 7
Figure 7. Figure 7: Analysis of reasoning prompt strategies. We report the effectiveness of Zero-Shot CoT, Self-Consistency, and Identifi￾cation CoT — across GPT-4o, Ovis2-34B, and InternVL2.5-38B under complete-view and partial-view settings. While CoT varia￾tions delivers notable gains …
Figure 8
Figure 8. Figure 8: Visualization of multi-view scene reconstruction and camera pose alignment. Although GPT-4o and Gemini-2.0-Flash both display moderate proficiency in single-view scene reconstruction, they struggle when aligning two different camera perspectives. Misidentifying camera …
Figure 9
Figure 9. Figure 9: Visualization of camera pose estimation. When asked to order the camera poses in clockwise order, MLLMs fail com￾pletely despite providing detailed reasonings. domain-specific training or architectural refinements, espe￾cially those emphasizing viewpoint consistency ma…
Figure 10
Figure 10. Figure 10: The streamlined version of annotation guideline for annotators to follow. It outlines key verification steps, common pitfalls, and examples to help annotators improve question clarity, accuracy, and answer quality. question benchmark, we invited two additional Ph.D. s…
Figure 11
Figure 11. Figure 11: Left: A structured JSON representation of a question-answer pair. Right: A snapshot of the GUI-based Annotation Platform used for reviewing and refining annotations. Best viewed zoomed in for details. What if the fried pan on the kitchen island is moved to the edge ne…
Figure 12
Figure 12. Figure 12: Comparison of a flagged ambiguous case before modification and its finalized version after cross-checking. The initial annotation was reviewed by multiple annotators, with ambi￾guities resolved through discussions to ensure clarity and consis￾tency. the post-prompt. T…
Figure 13
Figure 13. Figure 13: Our proposed Identification CoT prompt. To design for counting tasks with partial-visibility, our prompt guides the MLLM to systematically list each target entity across all views. tasks and Q&A pairs, demonstrating the diversity and com￾plexity of our benchmark. 9.2.…
Figure 14
Figure 14. Figure 14: Comparison of model outputs on the same questions under different prompting methods. (Case 1) [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Comparison of model outputs on the same questions under different prompting methods. (Case 2) [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: All-Angles Bench Samples (Part I) [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: All-Angles Bench Samples (Part II) [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Paired Data Samples (Part I) [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Paired Data Samples (Part II) [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Visualization of the reasoning process for two MLLMs (Part I). In this case, both MLLMs choose incorrect options due to errors in their reasoning process [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: Visualization of the reasoning process for two MLLMs (Part II). In the above case, GPT-4o selects the correct option but contain errors in its reasoning process. In the case below, both GPT-4o and Gemini-2.0-Flash follow a correct reasoning process and ultimately sele…
Figure 22
Figure 22. Figure 22: Questions that evaluators answered incorrectly, along with a detailed review of their reasoning for selecting the incorrect options. [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]
Figure 23
Figure 23. Figure 23: The visualization of all model performance across the 6 task categories in [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]
Figure 24
Figure 24. Figure 24: The prompt used for extracting multiple-choice answers from predictions. [PITH_FULL_IMAGE:figures/full_fig_p026_24.png]
Figure 25
Figure 25. Figure 25: The system prompt used for generating five tasks with the MLLM. [PITH_FULL_IMAGE:figures/full_fig_p026_25.png]
Figure 26
Figure 26. Figure 26: The task-specific prompts used for generating five tasks with the MLLM. [PITH_FULL_IMAGE:figures/full_fig_p027_26.png]
Figure 27
Figure 27. Figure 27: The question template designed for the Camera Pose Estimation task. [PITH_FULL_IMAGE:figures/full_fig_p028_27.png]
Figure 28
Figure 28. Figure 28: Visualization prompt designed to evaluate MLLMs on orientation-sensitive challenges. [PITH_FULL_IMAGE:figures/full_fig_p028_28.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpaceVista: All-Scale Visual Spatial Reasoning from mm to km

    cs.CV 2025-10 conditional novelty 7.0 of 10

    SpaceVista contributes a 1M-QA, 38K-video all-scale spatial reasoning dataset spanning mm to km, a manually verified benchmark, and a fine-tuned 7B MLLM with scale experts and progressive reward training.

  2. Seeing Isn't Orienting: A Cognitively Informed Hierarchical Benchmark for Object Orientation in MLLMs

    cs.CV 2025-05 unverdicted novelty 7.0 of 10

    DORI shows state-of-the-art multimodal models perform near-random on fine-grained object orientation reasoning despite strong scores on broader spatial benchmarks.

  3. Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence

    cs.AI 2026-08 conditional novelty 6.0 of 10

    An execution-centric training recipe that specializes a vision-language model into four capability experts and consolidates them via weight merging and distillation yields a single embodied model with broad benchmark gains.

  4. Cambrian-P: Pose-Grounded Video Understanding

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Cambrian-P adds per-frame camera pose tokens and a regression head to video MLLMs, delivering 4.5-6.5% gains on spatial benchmarks, generalization to other video QA tasks, and SOTA streaming pose estimation on ScanNet.

  5. BeliefNest: A Joint Action Simulator for Embodied Agents with Theory of Mind

    cs.AI 2025-05 conditional novelty 6.0 of 10

    BeliefNest represents nested beliefs as hierarchical Minecraft simulators and shows LLM agents can use them to pass false-belief tasks.

  6. Physical Plausibility Reasoning via HCM-GRPO: Empowering Compact Model for Superior Performance

    cs.CV 2025-11 conditional novelty 5.0 of 10

    A 2B multimodal model trained with HCM-GRPO, a GRPO variant with partial-credit rewards and hard-case oversampling, outperforms larger models on the authors' private physical-plausibility test set.

Reference graph

Works this paper leans on

88 extracted references · 52 canonical work pages · cited by 6 Pith papers

  1. [1]

    Claude, 2024

    Anthropic. Claude, 2024. 2, 4, 5

    2024

  2. [2]

    Digirl: Training in-the- wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the- wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Sys- tems, 37:12461–12495, 2025. 8

    2025

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. 2023. 8

    2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 4, 5

    arXiv 2025

  5. [5]

    Hourvideo: 1-hour video-language understanding

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Crist ´obal Eyzaguirre, Zane Du- rante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video-language understanding. Advances in Neural Information Processing Systems, 37:53168–53197, 2025. 8

    2025

  6. [6]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14455–14465,

  7. [7]

    Longvila: Scaling long-context visual language models for long videos

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 8

    arXiv 2024

  8. [8]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 4, 3

    arXiv 2024

Show all 88 references
  1. [9]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  2. [10]

    Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els. Advances in Neural Information Processing Systems , 37:135062–135093, 2025. 2

    2025

  3. [11]

    Sft memorizes, rl generalizes: A compara- tive study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A compara- tive study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025. 8

    2025 arXiv

  4. [12]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In NeurIPS, 2024. 8

    2024

  5. [13]

    Embodied question answer- ing

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018. 2

    2018

  6. [14]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023. 2, 8

    2023

  7. [15]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia...

    2024

  8. [16]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 8

    2024 arXiv

  9. [17]

    Gemini, 2023

    Google. Gemini, 2023. 2, 8

    2023

  10. [18]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings...

    2024

  11. [19]

    3d concept learn- ing and reasoning from multi-view images

    Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learn- ing and reasoning from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9202–9212, 2023. 2, 8

    2023

  12. [20]

    Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents. In Interna- tional conference on machine learning , pages 9118–9147. PMLR, 2022. 2

    2022

  13. [21]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 7, 8

    2024 arXiv

  14. [22]

    Conceptfusion: Open-set multimodal 3d mapping

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023. 2

    2023 arXiv

  15. [23]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pages 289–310. Springer, 2024. 2

  16. [24]

    Ego-humans: An ego- centric 3d multi-human benchmark

    Rawal Khirodkar, Aayush Bansal, Lingni Ma, Richard New- combe, Minh V o, and Kris Kitani. Ego-humans: An ego- centric 3d multi-human benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19807–19819, 2023. 2, 3, 4, 1

    2023

  17. [25]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 2

    2024 arXiv

  18. [26]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13299–13308, 2024. 8

    2024

  19. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 4, 8

    2024 arXiv

  20. [29]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  21. [30]

    Embodied agent inter- face: Benchmarking llms for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Er- ran Li Li, Ruohan Zhang, et al. Embodied agent inter- face: Benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems , 37: 100428–...

    2025

  22. [31]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024. 2

    2024

  23. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 8

    2023

  24. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,

  25. [34]

    Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024. 8

    2024

  26. [35]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts. In ICLR, 2023. 8

    2023

  27. [36]

    Ovis: Structural embed- ding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embed- ding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024. 4, 5, 6

    2024 arXiv

  28. [37]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023. 8

    2023

  29. [38]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In ACL, 2022. 8

    2022

  30. [39]

    Llarva: Vision-action instruction tuning enhances robot learning

    Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406.11815, 2024. 2

    2024 arXiv

  31. [40]

    gpt4o, 2024

    OpenAI. gpt4o, 2024. 2, 4, 6, 1, 3

    2024

  32. [41]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Confer- ence for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 2

    2020

  33. [42]

    Forgotten polygons: Multimodal large language models are shape-blind

    William Rudman, Michal Golovanesky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, and Ritambhara Singh. Forgotten polygons: Multimodal large language models are shape-blind. arXiv preprint arXiv:2502.15969, 2025. 2

    2025 arXiv

  34. [43]

    One step at a time: Long-horizon vision-and-language navigation with milestones

    Chan Hee Song, Jihyung Kil, Tai-Yu Pan, Brian M Sadler, Wei-Lun Chao, and Yu Su. One step at a time: Long-horizon vision-and-language navigation with milestones. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15482–15491, 2022. 2

    2022

  35. [44]

    Embodied bert: A trans- former model for embodied, language-guided visual task completion

    Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A trans- former model for embodied, language-guided visual task completion. arXiv preprint arXiv:2108.04927, 2021. 2

    2021 arXiv

  36. [45]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2, 4, 5, 7

    2023 arXiv

  37. [46]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. In NeurIPS, 2024. 4, 8

    2024

  38. [47]

    Metamorph: Multimodal understanding and generation via instruction tuning

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164, 2024. 8

    2024 arXiv

  39. [48]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024. 8

    2024

  40. [49]

    Mllm-tool: A multimodal large language model for tool agent learning

    Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Zhengxin Li, Lin Ma, Shenghua Gao, et al. Mllm-tool: A multimodal large language model for tool agent learning. arXiv preprint arXiv:2401.10727 ,

    arXiv

  41. [50]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023. 3

    2023

  42. [51]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. In NeurIPS, 2022. 2

    2022

  43. [52]

    Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024. 4

    2024 arXiv

  44. [53]

    Cad-mllm: Unifying multimodality- conditioned cad generation with mllm

    Jingwei Xu, Zibo Zhao, Chenyu Wang, Wen Liu, Yi Ma, and Shenghua Gao. Cad-mllm: Unifying multimodality- conditioned cad generation with mllm. arXiv preprint arXiv:2411.04954, 2024. 8

    2024 arXiv

  45. [54]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 1

    2024 arXiv

  46. [55]

    Thinking in space: How mul- timodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024. 2, 6, 7, 8

    2024 arXiv

  47. [56]

    Dynamic multimodal evalua- tion with flexible complexity by vision-language bootstrap- ping

    Yue Yang, Shuibai Zhang, Wenqi Shao, Kaipeng Zhang, Yi Bin, Yu Wang, and Ping Luo. Dynamic multimodal evalua- tion with flexible complexity by vision-language bootstrap- ping. arXiv preprint arXiv:2410.08695, 2024. 4

    2024 arXiv

  48. [57]

    Inst3d-lmm: Instance-aware 3d scene under- standing with multi-modal instruction tuning, 2025

    Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-lmm: Instance-aware 3d scene under- standing with multi-modal instruction tuning, 2025. 2

    2025

  49. [58]

    Multi-target embodied question answering

    Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra. Multi-target embodied question answering. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6309–6318, 2019. 2

    2019

  50. [59]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In CVPR, 2024. 8

    2024

  51. [60]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  52. [61]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning

    Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In NeurIPS, 2024. 8

    2024

  53. [62]

    Vlm 2-bench: A closer look at how well vlms im- plicitly link explicit matching visual cues

    Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, et al. Vlm 2-bench: A closer look at how well vlms im- plicitly link explicit matching visual cues. arXiv preprint arXiv:2502.12084, 2025. 2

    2025 arXiv

  54. [63]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 4

    2024

  55. [64]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 8

    2024 arXiv

  56. [65]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. arXiv preprint arXiv:2409.18125, 2024. 2

    2024 arXiv

  57. [66]

    Dynamic evaluation of large language models by meta probing agents

    Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. arXiv preprint arXiv:2402.14865,

    arXiv

  58. [68]

    Dataset Collection We manually selected 83 scenes from Ego-Exo4D [18] and 7 scenes from EgoHumans [24] to ensure the diversity of scenes

    Construction and Annotation Pipeline 7.1. Dataset Collection We manually selected 83 scenes from Ego-Exo4D [18] and 7 scenes from EgoHumans [24] to ensure the diversity of scenes. Given the high density of viewpoints in some Ego- Humans scenes, we carefully curated a subset of...

  59. [69]

    Answer with the option’s letter from the given choices directly

    Experiment Details 8.1. Evaluation Setup Our evaluation is conducted using the VLMEvalKit [15] framework. In order to ensure the reproducibility of our evaluation, we employ a greedy decoding strategy for all models, setting the temperature to 0 unless other- wise stated. The ...

  60. [70]

    Clarity & Relevance – The question must match the image/context and be easy to understand

  61. [71]

    Grammar & Precision – No errors or vague phrasing

  62. [72]

    What if the man with black pants in camera View 1 starts walking toward the view, what would be the trajectory of the man in camera View 3?

    Answer Quality – The correct answer must be factually accurate, and distractors should be plausible yet clearly incorrect. Example Before & After: Original Question: “What if the man with black pants in camera View 1 starts walking toward the view, what would be the trajectory...

  63. [73]

    Verify the questions and options for plausibility and accuracy based on the images/data provided

  64. [74]

    Correct unclear wording, grammatical errors, or inaccuracies in the questions

  65. [75]

    toward the view

    Finalize the correct answer. Common Mistakes to Avoid ✗ Vague phrasing (e.g., “toward the view”). ✗ Misleading or contradictory options. ✗ MLLM hallucinations (made-up facts or objects) Figure 10. The streamlined version of annotation guideline for annotators to follow.It outl...

  66. [76]

    While some questions do not require every view, we ensure that only the relevant ones are provided as input during inference

    Visualization Results In this section, we present a comprehensive visualization of the scene across all available views for convenience and consistency. While some questions do not require every view, we ensure that only the relevant ones are provided as input during inference...

  67. [77]

    Internally generate a detailed description of each visible person|note their appearance, clothing, orientation, movements, and any nearby detailed described objects or people

  68. [78]

    Cross-check these detailed descriptions across all views to avoid double-counting the same individual

  69. [79]

    Important Instructions: - You should perform your step-by-step reasoning privately and not reveal it to the user

    Arrive at the total number of unique people in the scene. Important Instructions: - You should perform your step-by-step reasoning privately and not reveal it to the user. Figure 13. Our proposed Identification CoT prompt. To design for counting tasks with partial-visibility, ...

  70. [80]

    Two seated individuals (2 people) consistently visible on the bench; 3

    Basketball player (1 person) appears in multiple images; 2. Two seated individuals (2 people) consistently visible on the bench; 3. Two standing individuals (2 people) seen in the background in at least one view. The same basketball player is visible across multiple images; Th...

  71. [81]

    The Stove: The stove is visible in all images and is a key reference point

    Identifying Common Elements: The Person: The person is the central figure in all images. The Stove: The stove is visible in all images and is a key reference point. The Kitchen Cabinets: The white kitchen cabinets are visible in all images and provide a backdrop. Tripods: Ther...

  72. [82]

    We can see another person in the background on the left

    Analyzing Relative Positions: View 1: This view is from the front-left of the person, showing their face and the stove from an angle. We can see another person in the background on the left. View 2: This view is from the left side of the person, showing the side of their face ...

  73. [83]

    Reconstructing the Layout (Top-Down View): Based on the relative positions, here's a sketch of the camera layout:

  74. [84]

    Backward

    Clockwise Order: Starting from the front-left (Image-1) and moving clockwise around the person: View 1 (Front-Left); View 2 (Left); View 4 (Back-Left); View 3 (Back) Therefore, the images in clockwise order are: View 1, View 2, View 4, View 3 View 1 (Front-Left) View 2 (Left) ...

  75. [85]

    Counting: Counting across multi-views Task Description: Could you count the amount on specific object (e.g., people, chair, cup) in the scene based on all the input camera views? Example: Question: Here are multiple camera views which are pointing to the same scene, could you ...

  76. [86]

    In View 1, there is a person in fencing who is facing to the spot where camera View 1 was positioned

    Attribute Identification: Attribute identification across multiple views Task Description: Could you identify the same object / attribute across multiple camera views? Example: Question: View 1 and View 2 are two different views that represent the same scene. In View 1, there ...

  77. [87]

    Relative Distance: Object-camera relative distance Task Description: Could you measure the relative distance between the seen object and camera view position in the cross view scenario? Example: Question: View 1 and View 2 are two different camera views that represent the same...

  78. [88]

    Relative Direction: Object-camera relative direction Task Description: Could you measure the relative direction between the seen object and camera view position in the cross view scenario? Example: Question: View 1 and View 2 are two different camera views that represent the s...

  79. [89]

    The task-specific prompts used for generating five tasks with the MLLM

    Manipulation: Relative object manipulation across views Task Description: Could you manipulate the seen objects (e.g., what if the person is walking to the left of the view), and design the Q&A based on the trajectory of object across views? Example: Question: What if the pers...

  80. [2024]

    4 Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs Supplementary Material In these supplementary materials, we provide the following: • Details on the construction and annotation pipeline of All- Angles Bench (Section 7); • Evaluation setup, implem...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.