Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training on InternSpatial, a 12-million-question open dataset with 19 spatial instruction formats, lifts an 8-billion-parameter VLM past proprietary frontier models on spatial benchmarks without hurting general performance.

desk verdict A genuinely large spatial-reasoning dataset with a new rotation-angle task, but the headline +12.1/+10.7 gains are confounded by the missing general-data-only control run. read the letter →

arxiv 2506.18385 v1 pith:DE56JLLH submitted 2025-06-23 cs.CV

Nianchen Deng , Lixin Gu , Shenglong Ye , Yinan He , Zhe Chen , Songze Li , Haomin Wang , Xingguang Wei
show 12 more authors
Tianshuo Yang Min Dou Tong He Wenqi Shao Kaipeng Zhang Yi Wang Botian Shi Yanting Zhang Jifeng Dai Yu Qiao Hongjie Zhang Wenhai Wang
This is my paper · ORCID
classification cs.CV
keywords spatialreasoningvision-languagemodelsdatasetbenchmarkmulti-viewrotationanglepredictioninstructionformats3Dsceneunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that spatial reasoning in vision-language models can be taught at scale by a large, open, automatically generated question-answer resource. It introduces InternSpatial, 12 million QA pairs spanning single-view and multi-view settings, plus InternSpatial-Bench, a diagnostic evaluation set in 19 instruction formats. A model fine-tuned on InternSpatial improves by 12.1 points on the authors' benchmark and by 10.7 points on the external VSI-Bench multi-view benchmark, while general-purpose scores stay roughly flat. If correct, this makes InternSpatial the largest open training resource of its kind and suggests that multi-view supervision and instruction-format diversity are the active ingredients. The authors also introduce a rotation-angle prediction task with 2.46 million pairs that has not appeared in prior spatial benchmarks.

What carries the argument

The load-bearing mechanism is the canonical view space: a 3D coordinate frame centered at the camera's optical center, with the y-axis along the viewing direction and the z-axis pointing upward, into which every object annotation is lifted. For 3D datasets this is a direct projection; for image-only datasets the pipeline estimates camera intrinsics, camera extrinsics, and dense metric depth, then lifts 2D boxes, masks, and captions into this frame so that spatial answers are computed geometrically. Around that frame, the data engine applies template-based QA generation, expands each QA into up to 19 textual and visual instruction formats, and adds multi-view tasks built from scanned scenes and object renderings, including the new rotation-angle prediction task. The geometry-grounded answers, not the templates themselves, are what the paper credits for the transfer.

What would settle it

Take a held-out set of images with known 3D ground truth, corrupt the estimated depth and camera parameters by increasing amounts, regenerate the InternSpatial QA labels from the corrupted geometry, and fine-tune identical models on each version; if benchmark gains fall as label noise rises, the claimed spatial learning is carried by the estimated geometry rather than by the dataset's design.

Watch

Extended reading notes

Core claim

The central claim is that a spatially capable VLM can be produced by supervised fine-tuning on InternSpatial: a fully automated pipeline takes images and 3D scans from diverse sources, lifts 2D object annotations into a canonical 3D view space, and converts them into templated questions and answers. The authors state that the resulting 8-billion-parameter model, trained with roughly 2 million sampled InternSpatial pairs plus general instruction data, outperforms the base model by 12.1 percentage points on InternSpatial-Bench and by 10.7 points on VSI-Bench, and beats the proprietary frontier models they compare against. They interpret this as evidence that scale, scene diversity, and 19 instruction formats teach transferable spatial competence rather than benchmark-specific answers.

Load-bearing premise

For image-only sources, the ground-truth answers are computed from estimated camera parameters and estimated depth; if those estimates are biased, the model learns the estimators' geometry rather than true spatial relations.

Editorial extensions

If this is right

  • Fine-tuning on InternSpatial raises the base 8-billion-parameter model by 12.1 points on InternSpatial-Bench, with the largest jumps in position comparison (+25.0) and size comparison (+20.9).
  • The same training transfers to the external multi-view benchmark VSI-Bench, lifting the average by 10.7 points with double-digit gains in object counting, object size estimation, and appearance order.
  • Training on all 19 instruction formats closes most of the format gap and also improves plain natural-language performance beyond training on raw questions alone.
  • General multimodal ability is preserved: five standard benchmarks change within roughly -1.6 to +1.8 points after spatial training.
  • The dataset introduces a rotation-angle prediction task with 2.46 million QA pairs, extending multi-view evaluation beyond existing benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the measured gains reflect genuine spatial understanding rather than template patterns, the dataset should transfer to downstream robotics tasks such as navigation and manipulation; the paper reports no such evaluation.
  • The image-only QA labels inherit errors from monocular depth and camera estimation, so filtering or reweighting samples by estimated label confidence could make training more robust.
  • A human-written, paraphrase-diverse holdout set would test whether models trained on templated questions generalize to free-form spatial language.
  • Extending the multi-view pipeline to egocentric video with measured camera trajectories could supply stronger supervision than estimated depth and test the same method's ceiling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces InternSpatial, a claimed 12M-pair spatial-reasoning dataset for VLMs spanning single-view and multi-view tasks, and InternSpatial-Bench, a 6,008-pair single-view benchmark, together with an extension of VSI-Bench with a rotation-angle prediction task. The dataset is built from COCO, AS-1B, Visual Genome, 3RScan, ScanNet, MultiScan, Cityscapes, Objaverse, and R2R, using a template-based QA generator with 19 instruction formats. The central empirical claim, in the abstract and in Tables 2 and 3, is that an InternVL2.5-8B model fine-tuned with InternSpatial gains 12.1 points on InternSpatial-Bench and 10.7 points on VSI-Bench relative to the untuned baseline while retaining general benchmark performance. The paper also claims a new multi-view rotation angle prediction task and reports that the trained model outperforms GPT-4o and Claude on the proposed benchmark.

Significance. If the causal attribution is correct, InternSpatial would be the largest open spatial-reasoning training resource for VLMs, and the reported VSI-Bench improvement would indicate transfer beyond the authors' own benchmark. The paper has strengths: a large and diverse set of sources, multi-view supervision including a large rotation-estimation component, full templates and prompts in the appendix, and a clearly specified training recipe. The external VSI-Bench gain is the most persuasive evidence for the dataset's utility. However, the internal benchmark is built with the same template and format-expansion machinery as the training data, and the main comparison lacks a general-data-only fine-tuning control, so the headline numbers cannot currently be read as a clean measurement of InternSpatial's effect. The resource's value to the community is real, but the empirical claims need re-grounding with additional controls and validation.

major comments (5)
  1. [Section 4.1, Tables 2 and 3] The reported +12.1 and +10.7 improvements compare InternVL-Spatial-8B, which is fine-tuned on a mixture of 291K general samples and InternSpatial samples, against the original untuned InternVL2.5-8B. No control model is trained on the 291K general samples alone. The deltas therefore conflate the effect of InternSpatial with the effect of an additional LoRA fine-tuning pass over general data. A matched general-data-only control is the minimal experiment needed to support the causal claim in the abstract, and it should also be applied to the multi-view VSI-Bench results.
  2. [Section 3.2 and Appendix D] InternSpatial-Bench is constructed with the same process family as the training set: Section 3.2 says instruction formats are expanded 'following the format extension strategy described in subsection 3.1', and Appendix D explicitly states that the benchmark 'reused the processes in the second and third stages of the training dataset pipeline' for answer generation and format expansion. In addition, the benchmark draws images from COCO, ScanNet, Objaverse, and Cityscapes test sets, whose training splits are all used in InternSpatial. The 12.1-point gain on InternSpatial-Bench is therefore partly a within-pipeline, within-distribution effect and should not be presented as an external generalization result. The paper should either construct a benchmark with independent question generation and answer verification, or explicitly reframe the internal result as a format/domain consistency check and place the burden of transfer evidence on VSI-Bench.
  3. [Section 3.1, 'Aligning to View Space'] For all image-only data sources, spatial ground truth is derived from estimated camera intrinsics, extrinsics, and dense depth using WildCamera, PerspectiveFields, and Metric3Dv2, with no reported validation of these estimates against ground truth and no audit statistics for the resulting QA labels. If these estimates are noisy, answers about near/far, above/below, and size are systematically wrong, and the model may learn the estimation models' biases rather than true spatial relations. The paper should report a validation study, for example comparing the estimated metric properties against known ground truth on a subset of ScanNet or similar data, and provide human-verification statistics for the training and benchmark QAs.
  4. [Section 4.4] The ablation model InternVL-Spatial-Raw-8B is described as trained on 'InternSpatial-Bench' without format expansion. Training on the evaluation benchmark is a form of leakage that invalidates the comparison and the conclusion about cross-format transfer. If this is a typo and the intended training set is InternSpatial (the training set), the text must be corrected; otherwise, the ablation should be redone using a training split that is disjoint from the evaluation benchmark.
  5. [Section 4.3] The VSI-Bench result is the strongest evidence of transfer, but the paper does not report whether any VSI-Bench test videos are drawn from or derived from the multi-view training sources ScanNet, MultiScan, R2R, or Objaverse, all of which are used to build InternSpatial's multi-view training data. A distributional-overlap analysis for VSI-Bench should be reported so that the +10.7 gain can be interpreted as genuinely external.
minor comments (6)
  1. [Section 3.2] The text first states that InternSpatial-Bench has 'a total of 5,300 high-quality examples', but the later statistics and Table 7 report 6,008 QA pairs; these numbers should be reconciled.
  2. [Table 4 and Section 4.5] The table heading says 'MathVision' while the text refers to 'MathVista [35]'; the benchmark name and reference should be made consistent and correct.
  3. [Table 3] The model names 'LLaV A-NeXT-Video-72B' and 'LLaV A-OneVision-72B' contain typographical spacing errors and should be corrected to 'LLaVA-NeXT-Video-72B' and 'LLaVA-OneVision-72B'.
  4. [Section 1] The claim that rotation angle prediction is a 'novel' multi-view task should be substantiated with a comparison to existing object pose and multi-view geometry work, since angle prediction is a standard task in those fields.
  5. [Appendix E] The statement that training uses 'approximately 291K general training samples' and '2M samples from InternSpatial, counted with repetition' should clarify whether these are unique samples or include repeated examples, since repetition affects the reported training mix.
  6. [Abstract and Section 3.3] The paper describes InternSpatial as 'open-source' but does not provide a dataset release URL or hosting repository; this should be included for reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

InternSpatial-Bench reuses the training pipeline's format expansion, making the +12.1 headline partly in-family; external VSI-Bench gain prevents full circularity.

  1. self definitional [Appendix D; Section 3.2 and Section 3.3 (InternSpatial-Bench construction)]
    "Subsequently, we reused the processes in the second and third stages of the training dataset pipeline to generate answers and expand the instruction formats."

    The evaluation benchmark is generated by the same format-expansion and answer-generation machinery as the training set. Section 3.2 says benchmark items are augmented 'following the format extension strategy described in subsection 3.1,' and Section 3.3 says InternSpatial-Bench 'appl[ies] the same instruction format expansion strategy as used in InternSpatial.' Appendix D makes the reuse explicit. Therefore the 12.1-point improvement on InternSpatial-Bench partly reflects the model's fit to the training pipeline's templates and visual formats (box, mask, number annotations) rather than a newly measured spatial skill. This is a same-family evaluation: the benchmark's format distribution is an input to the training data construction.

full rationale

The dataset construction itself is not circular: for 3D sources, spatial ground truth comes from provided global annotations and camera parameters, and for image-only sources it is lifted through external estimators (WildCamera, PerspectiveFields, Metric3Dv2), not from the model being trained. The principal circularity concern is the benchmark. InternSpatial-Bench reuses the training pipeline's format-expansion and answer-generation stages, so the reported 12.1-point gain on InternSpatial-Bench is partly an in-family evaluation. However, the 10.7-point gain on VSI-Bench is an external benchmark and provides independent evidence that the training transfers beyond the authors' own evaluation machinery. Separately, the experimental setup lacks a control trained on the 291K general samples alone, so causal attribution of the full delta to InternSpatial is underdetermined; that is a correctness/confound issue rather than a circularity step under the rubric. Overall, the central claim retains independent content, and the circularity is partial rather than total.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a chain of unverified annotation layers: VLM detections, SAM2 masks, and monocular depth and camera estimation for image-only sources, plus source dataset annotations for 3D sources. The benchmark adds a manual verification layer, but the training set is not audited and no release artifacts are provided. There are no fitted constants in a derivation sense; the listed free parameters are hand-chosen pipeline thresholds and the 2M training budget behind the reported gains.

free parameters (2)
  • InternSpatial training sample budget = 2M QA pairs from 12M, counted with repetition
    The reported gains use a 2M subsample of the dataset; no ablation shows sensitivity to this budget, so it is a hand-chosen variable in the headline results.
  • Oriented bounding box inclusion threshold = principal dimension > 15 cm
    Objects below this size are handled through existing annotations to avoid ambiguity; the threshold is chosen by hand in Section 3.1 and is not validated.
assumptions (5)
  • domain assumption Monocular depth, intrinsic, and extrinsic estimates from Metric3Dv2, WildCamera, and PerspectiveFields are accurate enough to define ground-truth spatial relations in image-only datasets.
    Section 3.1, Aligning to View Space: single-view QAs depend on lifting 2D annotations into a canonical 3D view space using these estimates; errors propagate directly into answers.
  • domain assumption Qwen2.5-VL 72B detections, captions, and grounding, plus SAM2 masks, provide reliable object-level annotations for millions of images.
    Section 3.1 Generating Image-level Annotation uses an open-source VLM and SAM2 when annotations are missing; no quality audit of the training-set annotations is reported.
  • domain assumption Template-based question generation with filtering and positive/negative balancing yields correct, diverse QA pairs at scale.
    Section 3.1 Template-based QA Generation; filtering rules are described qualitatively and the paper acknowledges template language may underrepresent natural richness in Section 5.
  • domain assumption The 3D datasets' global annotations, camera parameters, room centroids, and alpha-shape room sizes are correct.
    Section 3.1 relies on ScanNet, MultiScan, R2R, and Objaverse annotations and on Alpha Shape and Open3D processing without independent verification.
  • domain assumption Manual verification of InternSpatial-Bench QA pairs is sufficient to guarantee benchmark correctness.
    Appendix D states human annotators verified all benchmark pairs; the number of annotators, instructions, and inter-annotator agreement are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models." pith.science (2026). https://pith.science/paper/DE56JLLH

@misc{pith2026250618385,
  author       = {Pith},
  title        = {Pith review of: InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DE56JLLH}},
  note         = {Machine review of arXiv:2506.18385}
}
read the original abstract

Recent benchmarks and datasets have been proposed to improve spatial reasoning in vision-language models (VLMs), yet existing open resources remain limited in scale, visual diversity, and instruction expressiveness. In this work, we introduce InternSpatial, the largest open-source dataset for spatial reasoning in VLMs, along with InternSpatial-Bench, a corresponding evaluation benchmark designed to assess spatial understanding under diverse instruction formats. InternSpatial comprises 12 million QA pairs spanning both single-view and multi-view settings, drawn from diverse visual environments and supporting 19 instruction formats that reflect varied query styles. For evaluation, we propose InternSpatial-Bench for single-view tasks and expand multi-view reasoning by introducing a novel rotation angle prediction task that has not been explored in prior work. Experimental results show that models trained on InternSpatial achieve 12.1% improvement on InternSpatial-Bench and 10.7% on VSI-Bench, while maintaining strong performance on general-purpose benchmarks. We hope these resources will support the development of spatially capable VLMs in practical applications such as robotics and embodied AI.

Figures

Figures reproduced from arXiv: 2506.18385 by the authors.

Figure 1
Figure 1. Generation pipeline for InternSpatial. The optional flows (represented by dashed lines and boxes) are only [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Examples of diverse instruction formats in text and image. The four images illustrate different visual formats: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of instruction formats (Left) and data sources (Right) in InternSpatial. Statistics of InternSpatial-Bench Following Spatial-Bench and Spatial-RGPT, our proposed benchmark, InternSpatial-Bench, includes five tasks—Position Estimation, Size Estimation, Rotation Estimation, Existence Es￾timation, and Object Counting—designed to systematically evaluate the spatial reasoning capabilities of VLMs. In total, … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of instruction formats (Left) and data sources (Right) in InternSpatial-Bench. 4 Experiments We begin in Section 4.1 by introducing the baseline model and outlining the evaluation benchmarks used in our experiments. Section 4.2 then presents results on Int…
Figure 5
Figure 5. Figure 5: The results of the different image (Left) and text (Right) formats in the ablation study. baseline InternVL2.5-8B on general reasoning benchmarks. Specifically, InternVL-Spatial-8B shows a performance gain of +1.8% on MathVista [35], -0.1% on OCRBench [36], +0.9% on Te…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations

    cs.CV 2026-03 conditional novelty 6.5 of 10

    Sparse multi-view reasoning is largely unsolved for VLMs; grounded CoT with visual evidence improves moderate cases and transfers to real data, but deep multi-hop reasoning still scales poorly.

  2. Transferability Between Understanding and Generation in Unified Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Cross-task capability transfer in UMMs is architecture-dependent and can be exploited by training understanding to improve generation while avoiding distribution shift.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    Spatialbot: Precise spatial understanding with vision language models

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. In2025 IEEE International Conference on Robotics and Automation (ICRA), 2025

    work page 2025

  2. [2]

    Spatialrgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 135062–135093. Curran As...

    work page 2024

  3. [3]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455–14465, June 2024

    2024

  4. [4]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces, 2024

    work page 2024

  5. [5]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014

    2014

  6. [6]

    The all-seeing project: Towards panoptic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. InICLR, 2024

    work page 2024

  7. [7]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    2017

  8. [8]

    Rio: 3d object instance re-localization in changing indoor environments

    Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Niessner. Rio: 3d object instance re-localization in changing indoor environments. InProceedings IEEE International Conference on Computer Vision (ICCV), 2019

    2019

Show all 53 references
  1. [9]

    Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niessner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niessner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  2. [10]

    Multiscan: Scalable rgbd scanning for 3d environments with articulated objects

    Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel Chang, and Manolis Savva. Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Sys...

    2022

  3. [11]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProc. of the IEEE Conference on Computer Vision and Pattern Recognition...

    2016

  4. [12]

    Objaverse: A universe of annotated 3d objects.arXiv preprint arXiv:2212.08051, 2022

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects.arXiv preprint arXiv:2212.08051, 2022

    2022 arXiv

  5. [13]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE Conference ...

    2018

  6. [14]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 2020

    2020

  7. [15]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  8. [16]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  9. [17]

    Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks

    Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. InCVPR, 2022. 10 A PREPRINT

    2022

  10. [18]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  11. [19]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  12. [20]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.NeurIPS, 2023

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.NeurIPS, 2023

    2023

  13. [21]

    Interngpt: Solving vision-centric tasks by interacting with chatbots beyond language

    Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Interngpt: Solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint arXiv:2305.05662, 2023

    2023 arXiv

  14. [22]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  15. [23]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  16. [24]

    3d-llm: Injecting the 3d world into large language models.NeurIPS, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.NeurIPS, 2023

    2023

  17. [25]

    Tenenbaum, and Chuang Gan

    Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  18. [26]

    Is a picture worth a thousand words? delving into spatial reasoning for vision language models

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editor...

    2024

  19. [27]

    Topviewrs: Vision-language models as top-view spatial reasoners, 2024

    Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, and Ivan Vuli´c. Topviewrs: Vision-language models as top-view spatial reasoners, 2024

    2024

  20. [28]

    Proximity qa: Unleashing the power of multi-modal large language models for spatial proximity analysis, 2024

    Jianing Li, Xi Nan, Ming Lu, Li Du, and Shanghang Zhang. Proximity qa: Unleashing the power of multi-modal large language models for spatial proximity analysis, 2024

    2024

  21. [29]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

    2024 arXiv

  22. [30]

    Tame a wild camera: In-the-wild monocular camera calibration

    Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu. Tame a wild camera: In-the-wild monocular camera calibration. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 45137–45149...

    2023

  23. [31]

    Linyi Jin, Jianming Zhang, Yannick Hold-Geoffroy, Oliver Wang, Kevin Blackburn-Matzen, Matthew Sticha, and David F. Fouhey. Perspective fields for single image camera calibration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages...

    2023

  24. [32]

    Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation.IEEE Transactions on Pattern Analysis ...

    2024

  25. [33]

    Akkiraju, H

    N. Akkiraju, H. Edelsbrunner, M. Facello, P. Fu, and C. Varela. Alpha shapes: Definition and software. InGCG: International Computational Geometry Software Workshop, 1995

    1995

  26. [34]

    Open3D: A modern library for 3D data processing

    Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018

    2018 arXiv

  27. [35]

    Measuring multimodal mathematical reasoning with math-vision dataset, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset, 2024

    2024

  28. [36]

    On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023

    Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023

    2023 arXiv

  29. [37]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. pages 8317–8326, 2019. 11 A PREPRINT

    2019

  30. [38]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. pages 2263–2279, 2022

    2022

  31. [39]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  32. [40]

    Gpt-4o system card.https://openai.com/index/gpt-4o-system-card/, 2025

    OpenAI. Gpt-4o system card.https://openai.com/index/gpt-4o-system-card/, 2025

    2025

  33. [41]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: //github.com/open-compass/opencompass, 2023

    2023

  34. [42]

    The claude 3 model family: Opus, sonnet, haiku.https://www.anthropic.com, 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku.https://www.anthropic.com, 2024. [43]https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

    2024

  35. [44]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025

  36. [45]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv prep...

    2024 arXiv

  37. [46]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. pages 26689–26699, 2024

    2024

  38. [47]

    Llava-next: A strong zero-shot video understanding model

    Y Zhang, B Li, H Liu, Y Lee, L Gui, D Fu, J Feng, Z Liu, and C Li. Llava-next: A strong zero-shot video understanding model. 2024

    2024

  39. [48]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  40. [49]

    q u e s t i o n _ t e m p l a t e s

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Represen- tations, 2022. 12 A PREPRINT Appendix A Explanation and Statistics...

    2022

  41. [50]

    5 - or - more - objects : There are at least 5 objects in the image

  42. [51]

    cartoon : This is a cartoon image

  43. [52]

    group - of - images : This is a group of images

  44. [53]

    s c r e e n s h o t : This is a s c r e e n s h o t

  45. [54]

    5 - or - more - objects

    realistic - image : This is a r ea li st ic image captured by camera . For each condition , answer true of false . Response in JSON dict with five fields : " 5 - or - more - objects " , " cartoon " , " group - of - images " , " s c r e e n s h o t " , " realistic - image " " "...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.