Pith. sign in

REVIEW 3 major objections 5 minor 8 cited by

Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VeBrain argues that robot control can be expressed as ordinary text-based multimodal LLM tasks in 2D image space—keypoint detection plus embodied skill recognition—so one model can hold multimodal understanding, spatial reasoning, and…

desk verdict The keypoint+skill control formulation is genuinely interesting, but the spatial benchmark gains look contaminated by ScanNet train/eval overlap, and the robot numbers are too small to carry the unified-claim. read the letter →

arxiv 2506.00123 v1 pith:IOYHG2TW submitted 2025-05-30 cs.CV cs.RO

Gen Luo , Ganlin Yang , Ziyang Gong , Guanzhou Chen , Haonan Duan , Erfei Cui , Ronglei Tong , Zhi Hou
show 10 more authors
Tianyi Zhang Zhe Chen Shenglong Ye Lewei Lu Jingbo Wang Wenhai Wang Jifeng Dai Yu Qiao Rongrong Ji Xizhou Zhu
This is my paper
classification cs.CVcs.RO
keywords visualembodiedintelligencemultimodallargelanguagemodelsvisual-spatialreasoningrobotcontrolkeypointdetectionskillrecognitioninstructiontuningchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that one multimodal large language model can be simultaneously good at everyday image understanding, spatial reasoning about 3D scenes, and controlling real robots, capabilities that usually pull in opposite directions. Its solution is to stop training robots with special action outputs and instead phrase every robot command as an ordinary text answer in a 2D image: the model outputs where the robot should go (a keypoint) and which skill to perform (a named action). A bolt-on robotic adapter converts these text signals into actual motion. If the claim holds, embodied AI no longer needs separate control models, and one generalist model can be dropped into a legged robot or arm without losing its general vision abilities.

What carries the argument

The load-bearing mechanism is the reformulation of robot control into two text-native subtasks in 2D visual space—keypoint detection, which names the pixel target the robot should approach or touch, and embodied skill recognition, which names one action from a fixed policy pool (walk, turn, grasp, dump, pull, and others). Because both outputs are ordinary text tokens, control and perception share one next-token objective in one vocabulary, which the paper argues removes the negative interference that plagues VLA training. A four-module robotic adapter (point tracker, movement controller, skill executor, dynamic takeover) then bridges text decisions to physical motion: LocoTrack keeps the 2D keypoints aligned as the ego camera moves, the movement controller projects them to 3D via depth and calibration, the skill executor calls the named policy, and dynamic takeover returns control to the MLLM when tracking fails or a subtask ends. The second pillar is VeBrain-600k, a 600k-item instruction dataset whose multimodal chain-of-thought conversations interleave perception, spatial reasoning, and control decisions so that one fine-tuning run teaches all three capabilities.

What would settle it

Re-run the five spatial benchmarks (ScanQA, SQA3D, ScanRefer, Multi3DRef, and VSI-Bench) with VeBrain trained on a version of VeBrain-600k from which all ScanNet-derived and GPT4Scene-derived items are removed, and compare against the published numbers; if the CIDEr and accuracy gains over Qwen2.5-VL largely disappear, the spatial-intelligence claim reduces to scene familiarity. Similarly, if the robot success rates drop sharply when object layouts, lighting, or camera angles differ from the 80-hour human-collected episodes, the control generalization claim fails.

Watch

Extended reading notes

Core claim

VeBrain's central claim is that the apparent conflict between multimodal understanding and robot control disappears once control is rewritten as an ordinary text-based MLLM task in the 2D image plane. Instead of predicting continuous action vectors as vision-language-action models do, the model outputs two textual quantities at each decision step: a keypoint (the pixel location the robot should move to or interact with) and an embodied skill (a named primitive such as walk, grasp, or dump). A robotic adapter—point tracker, movement controller, skill executor, and dynamic takeover—converts these signals into motion and hands control back to the MLLM when a target is lost. Trained on VeBrain-600k, a 600k-item instruction set that mixes multimodal understanding, spatial reasoning, and control data with multimodal chain-of-thought, the resulting 7B model reports the highest average on 13 multimodal plus 5 spatial benchmarks among the open MLLMs compared, including +5.6 on MMVet over its Qwen2.5-VL-7B backbone, and higher robot success rates than VLA baselines on legged and arm tasks, with +42.9 average success points over the π0 baseline on the arm benchmark.

Load-bearing premise

The spatial performance is measured on benchmarks built from the same scene corpus, ScanNet, that supplied a large part of the spatial training data, so the reported spatial gains could come from seeing those scenes during training rather than from real spatial generalization.

Editorial extensions

If this is right

  • A single instruction-tuned MLLM can hold general multimodal understanding and real-robot control in one parameter set, so embodied systems no longer need a separate VLA model.
  • Because control signals are text keypoints and skill names, the same model transfers across platforms by swapping only the adapter's low-level policy pool.
  • The task-specific chain-of-thought data is what the paper credits for compositional long-horizon behaviors such as opening a drawer, finding a hidden object, and then grasping it.
  • The paper's ablations imply that adding control data does not erode multimodal performance when control is formulated as text tasks, unlike action-policy training that drops MLLM scores.
  • If the framework holds, scaling VeBrain-600k with more control episodes should improve robot success rates without requiring a separate action decoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because a large share of the spatial training data is drawn from ScanNet and the spatial benchmarks are also ScanNet-based, the reported spatial gains should be read with caution until tested on non-overlapping scenes; this is our concern, not the paper's.
  • The keypoint-and-skill formulation is a discrete abstraction that could extend to other embodiments, such as drones, humanoids, or mobile manipulators, as long as each has a named skill pool; the paper does not test this.
  • The claimed unification is really at the interface level: the MLLM depends on a separate point tracker and a hand-built policy pool, so the system's generality hinges on components outside the model itself.
  • A useful extension would be to measure the three capabilities as a function of data proportion and model size, turning the reported ablation trade-offs into a scaling recipe for embodied MLLMs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VeBrain, a framework that unifies multimodal understanding, visual-spatial reasoning, and robot control in a single MLLM. Robot control is reformulated as keypoint detection and embodied skill recognition in 2D, and a robotic adapter converts these textual signals into executable motion policies. The authors introduce VeBrain-600k, a 600k-item instruction dataset mixing the three capabilities via multimodal chain-of-thought, and report gains over Qwen2.5-VL on several multimodal and spatial benchmarks, along with success-rate improvements over VLA baselines on legged and arm tasks.

Significance. The framework idea is appealing: treating control as text-based MLLM tasks in 2D space and closing the loop with a robotic adapter is a clean way to reuse MLLM training for embodiment, and the VeBrain-600k dataset is a substantial public resource. If the reported results were valid, the paper would be a worthwhile step toward unified embodied MLLMs. However, the evaluation contains multiple confounds that prevent the current experiments from supporting the central claims: possible train/eval scene overlap in the spatial benchmarks, a data-scale difference behind the multimodal gains, and an in-distribution, asymmetric robot-control evaluation. These issues are load-bearing for the paper's headline assertions.

major comments (3)
  1. [Sec. 3.3 and App. A.2] The visual-spatial training data consists of 155k items from GPT4Scene plus 157k self-collected video data generated from ScanNet (31.4k unique items repeated five times). The four 3D benchmarks in Tab. 4 (ScanQA, SQA3D, ScanRefer, Multi3DRef) are all constructed from ScanNet scenes, and VSI-Bench also contains ScanNet-derived videos. The paper never reports whether the scan IDs used for training are disjoint from the validation scan IDs used for evaluation. If they overlap, the large spatial gains (e.g., ScanQA CIDEr jumping from 62.7 to 101.5 in Tab. 4, and from 56.8 to 102.2 in Tab. 1 after adding spatial data) would reflect memorization of scene layouts rather than generalized spatial reasoning. This directly threatens the central claim of superior spatial intelligence; the authors should provide a scene-disjoint split or re-evaluate on scenes never seen in training.
  2. [Tab. 1 and Tab. 3] The +5.6 improvement on MMVet (67.1 to 72.7) appears only after adding the 200k multimodal understanding conversations; the framework itself, with control data and spatial reasoning data, leaves MMVet at 64.7, below the Qwen2.5-VL baseline. Consequently, the comparison between VeBrain and Qwen2.5-VL in Tab. 3 does not isolate the effect of the proposed framework, because VeBrain is trained on substantially more data. The claim that VeBrain outperforms existing MLLMs on multimodal benchmarks is therefore not supported as stated; it could be a data-scale effect. The authors should add a baseline trained on the same 200k multimodal conversations (without spatial and control data) or otherwise control for training data scale.
  3. [Sec. 4.1, App. A.2, and App. A.4] The robot-control evaluation uses tasks built from the same self-collected scenes and task definitions used to create the 88k control items in VeBrain-600k. Tab. 7 further states that OpenVLA and pi0 are allowed only 10 demonstrations per task, whereas VeBrain is trained on thousands of in-domain examples from the same setup. With 10 trials per task and no error bars, Tabs. 6-7 measure in-distribution execution rather than generalization, and the comparisons are asymmetric. This undermines the claims of strong adaptability and the +42.9% average success rate over pi0. The authors should evaluate on new scenes or tasks, report variance or more trials, and match the amount of in-domain training data across methods.
minor comments (5)
  1. [Abstract and Tab. 6] The abstract states '+50% average gains' on legged robots, but Tab. 6 shows VeBrain at 86.4 vs Qwen2.5-VL at 42.1, which is approximately +44 percentage points or +105% relative; the wording is imprecise.
  2. [Tab. 5] VeBrain's average VSI-Bench gain hides declines on several subcategories (Rel. Dir, Router, Plan, Appr., Order) compared to Qwen2.5-VL; the paper should discuss these discrepancies rather than only reporting the average.
  3. [Sec. 3.2.2 and App. A.3.1] The tracker is called 'LocoTrack' in Sec. 3.2.2 and 'Locotrack-small' in App. A.3.1; please unify the name and reference.
  4. [Fig. 1] The text 'Diverse Data Sources' is rendered as 'Dirverse Data Sources'; please fix the typo.
  5. [References] Several reference entries contain '- ' placeholders for missing pages or venue details (e.g., [15], [20], [97]); please normalize the bibliography.

Circularity Check

1 steps flagged · score 4.0 of 10

Robot-control evaluation is performed in the same self-collected scenes used to build VeBrain-600k, so the reported control success is partly a fit to the training setup; the spatial benchmarks share ScanNet with the training data and no disjoint split is reported.

  1. other [Sec 3.3 (Data collection) and Sec 4.1 (Evaluation Datasets); Tabs 6-7]
    "For robot control, we collect a multimodal robot dataset with 88k items from scratch, including locomotion and manipulation of legged robots and robotic arms. ... Then, 5 human experts manually annotate the keypoints and actions of these episodes. ... For robot control, evaluation benchmarks are built from our self-collected scenes, as described in the appendix."

    The 88k training episodes and the evaluation benchmarks are drawn from the same self-collected scenes, and the evaluation tasks (Table 10) use the exact skill and keypoint annotations on which the model was fine-tuned. The success rates in Tabs 6-7 therefore measure in-distribution reproduction of the training setup, not generalization to held-out scenes or tasks. The claim of 'strong adaptability, flexibility, and compositional capabilities' is not independently tested.

full rationale

The core formulation (keypoint detection plus skill recognition as text tasks) is a design choice and is not circular in itself: the robotic adapter and the closed loop are new components, and the multimodal benchmarks (MMVet, TextVQA, etc.) are external and not built from the training data. The circularity is in the evaluation, not the mathematics. For robot control, the paper states that evaluation benchmarks are built from the same self-collected scenes that produced the 88k training items, so the reported control results are partly forced by the training distribution. For spatial reasoning, the 157k self-collected training videos are generated from ScanNet and the four 3D benchmarks (ScanQA, SQA3D, ScanRefer, Multi3DRef) are also ScanNet-based; the paper does not report whether the scene IDs used for generating training video are disjoint from the validation scenes, so the spatial gains cannot be confirmed as generalization rather than scene familiarity. This missing split is a support gap, not a demonstrated identity, and the multimodal results remain independent evidence. Overall, the central 'generalization' claim is partially circular, but the framework itself is not a derivation-equivalent to its inputs.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the availability of pre-trained skills, reliable point tracking, accurate depth conversion, and implicitly on the disjointness of training and evaluation scenes. The data mixture ratio was tuned on the same benchmark family, so it is a free parameter rather than a principled choice.

free parameters (1)
  • Data mixture proportions for VeBrain-600k = 200k multimodal, 312k spatial, 88k robot
    Chosen by ablation on the same benchmark suite (Table 11), so the balance is tuned to the evaluation metrics rather than derived from first principles.
assumptions (5)
  • domain assumption Pre-trained low-level policies in the policy pool are sufficient to execute the skills the MLLM names.
    Sec 3.2.2 and Table 9 assume walking, turning, dumping, grasping, and similar skills are available and reliable; failures are handled by the takeover mechanism, but the pool coverage is an input.
  • domain assumption The point tracker (LocoTrack) reliably maintains 2D keypoints under ego-motion.
    Sec 3.2.2 relies on the tracker to update keypoints as the robot moves; if tracking fails, the closed-loop control breaks.
  • domain assumption RGBD calibration and 2D-to-3D conversion are accurate for the tested scenes.
    Appendix A.3 describes the transformation but does not report calibration error or robustness across scenes.
  • domain assumption GPT-4o and Gemini generated chain-of-thought is a valid supervision signal for spatial reasoning and control.
    Sec 3.3 uses LLM-generated CoT as training targets; the paper validates only a 10% random sample and does not measure how often the CoT is factually correct.
  • domain assumption ScanNet-derived training data does not overlap with the ScanQA, SQA3D, ScanRefer, and Multi3DRef evaluation scenes.
    The paper does not report a scene-level split. The training data is explicitly from ScanNet (Sec 3.3), and the evaluation benchmarks are also ScanNet-based (Sec 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces." pith.science (2026). https://pith.science/paper/IOYHG2TW

@misc{pith2026250600123,
  author       = {Pith},
  title        = {Pith review of: Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOYHG2TW}},
  note         = {Machine review of arXiv:2506.00123}
}
read the original abstract

The remarkable progress of Multimodal Large Language Models (MLLMs) has attracted increasing attention to extend them to physical entities like legged robot. This typically requires MLLMs to not only grasp multimodal understanding abilities, but also integrate visual-spatial reasoning and physical interaction capabilities. Nevertheless,existing methods struggle to unify these capabilities due to their fundamental differences.In this paper, we present the Visual Embodied Brain (VeBrain), a unified framework for perception, reasoning, and control in real world. VeBrain reformulates robotic control into common text-based MLLM tasks in the 2D visual space, thus unifying the objectives and mapping spaces of different tasks. Then, a novel robotic adapter is proposed to convert textual control signals from MLLMs to motion policies of real robots. From the data perspective, we further introduce VeBrain-600k, a high-quality instruction dataset encompassing various capabilities of VeBrain. In VeBrain-600k, we take hundreds of hours to collect, curate and annotate the data, and adopt multimodal chain-of-thought(CoT) to mix the different capabilities into a single conversation. Extensive experiments on 13 multimodal benchmarks and 5 spatial intelligence benchmarks demonstrate the superior performance of VeBrain to existing MLLMs like Qwen2.5-VL. When deployed to legged robots and robotic arms, VeBrain shows strong adaptability, flexibility, and compositional capabilities compared to existing methods. For example, compared to Qwen2.5-VL, VeBrain not only achieves substantial gains on MMVet by +5.6%, but also excels in legged robot tasks with +50% average gains.

Figures

Figures reproduced from arXiv: 2506.00123 by the authors.

Figure 1
Figure 1. Overview of VeBrain and VeBrain-600k. Compared to existing MLLMs, VeBrain achieves the best trade-off performance on benchmarks of multimodal understanding, visual-spatial reasoning, and robot control into one MLLM. To support the unified training of VeBrain, VeBrain-600k is built with a semi-automated data engine covering a variety of data sources and tasks. understanding capability [110]. To compensate for this sh… view at source ↗
Figure 2
Figure 2. Illustration of VeBrain architecture and robotic adapter. In VeBrain, the MLLM is capable of perception, thinking, and decision-making in common MLLM tasks. For robot control, an additional adapter is combined with the MLLM to achieve closed-loop control of the real robot. fine-tuning on robotic interaction data, VLAs [86, 45, 88, 10, 32, 40] process visual observations and textual instructions through dedicated enc… view at source ↗
Figure 3
Figure 3. Visualization of VeBrain on robotic arm and legged robot. VeBrain demonstrates compositional capabilities in handling complex robotic tasks. Due to space limitations, most of the text regarding the thinking process is omitted in the figure. size of VeBrain than closed-source MLLMs, these results further confirm the strong multimodal abilities of VeBrain. Results of visual-spatial reasoning. Tab. 4 first demonstrates… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Dense per-frame intermediate representations (traces, masks, grasp poses, subtasks) improve embodied VQA, VLA action generation, and world-model video prediction in the new 230k-episode RoboInter-Data suite.

  2. ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A single 8B backbone unifies spatial perception, decision making, navigation/manipulation, and progress estimation with SSR+ merging, reporting gains on most spatial benchmarks and competitive action/progress results.

  3. RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thought

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    Training a 4B vision-language model to emit identity-tracked, visually grounded reasoning anchors improves embodied spatial, multi-view, and pointing task performance over 7B baselines.

  4. RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    RoboProcessBench is a new benchmark decomposing process-aware understanding into static monitoring and dynamic reasoning across 12 question families, with evaluations showing VLM limitations but post-training gains on...

  5. Contrastive Representation Regularization for Vision-Language-Action Models

    cs.RO 2025-10 conditional novelty 6.0 of 10

    Adding a robot-state-aware contrastive loss to VLA training improves manipulation success on RoboCasa-Kitchen and real-robot tasks.

  6. RoboBrain 2.0 Technical Report

    cs.RO 2025-07 conditional novelty 6.0 of 10

    RoboBrain 2.0, a 7B/32B embodied vision-language model built on Qwen2.5-VL, reports state-of-the-art or near-top scores on several spatial and temporal reasoning benchmarks for robotics.

  7. Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

    cs.RO 2026-06 unverdicted novelty 5.0 of 10

    Embodied-R1.5 is an 8B EFM achieving SOTA on 16 of 24 embodied VLM benchmarks, fine-tunable to outperform leading VLAs, with claimed zero-shot real-robot generalization.

  8. The high-speed X-ray camera on AXIS: design and performance updates

    astro-ph.IM 2025-08 unverdicted novelty 4.0 of 10

    An X-ray camera design-update whose supporting full text is a different paper (RynnEC, an embodied AI model), leaving all camera performance claims unverified.

Reference graph

Works this paper leans on

112 extracted references · 23 canonical work pages · cited by 8 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 3

    arXiv 2023

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022. 1

    arXiv 2022

  3. [3]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binko...

    2022

  4. [4]

    Claude 3.5 sonnet.https://www.anthropic.com/news/claude-3-5-sonnet, 2024

    Anthropic. Claude 3.5 sonnet.https://www.anthropic.com/news/claude-3-5-sonnet, 2024. 7 10

    2024

  5. [5]

    Openflamingo: An open-source framework for training large autoregressive vision-language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023. 3

    arXiv 2023

  6. [6]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022. 1, 7

    2022

  7. [7]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  8. [8]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923 , 2025. 1, 3, 4, 5, 6, 7, 8, 9, 10, 18, 20

    arXiv 2025

Show all 112 references
  1. [9]

    arXiv preprint arXiv:2410.24164 , 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 , 2024. 1, 2, 9, 10

    arXiv

  2. [10]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817 , 2022

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817 , 2022. 1, 4

    2022 arXiv

  3. [11]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818 , 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818 , 2023. 4

    2023 arXiv

  4. [12]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020. 3

    1901

  5. [13]

    Com- monsense reasoning for legged robot adaptation with vision-language models.arXiv preprint arXiv:2407.02666 ,

    Annie S Chen, Alec M Lessing, Andy Tang, Govind Chada, Laura Smith, Sergey Levine, and Chelsea Finn. Com- monsense reasoning for legged robot adaptation with vision-language models.arXiv preprint arXiv:2407.02666 ,

    arXiv

  6. [14]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InEuropean conference on computer vision , pages 202–221. Springer, 2020. 7

    2020

  7. [15]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier vision-language models. arXiv preprint arXiv:2504.15271 , 2025. 7

    2025

  8. [16]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195 , 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195 , 2023. 3

    2023 arXiv

  9. [17]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision , pages 370–387. Springer, 2024. 6, 19

    2024

  10. [18]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024. 7

    2024 arXiv

  11. [19]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv: 2312.14238 , 2023

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv: 2312.1...

    2023 arXiv

  12. [20]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271 , 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271 , 2024. 7

    2024 arXiv

  13. [21]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv:2404.16821, 2024. 1, 3

    2024 arXiv

  14. [22]

    Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024. 4

    2024 arXiv

  15. [23]

    Spatially-aware transformer for embodied agents.arXiv preprint arXiv:2402.15160, 2024

    Junmo Cho, Jaesik Yoon, and Sungjin Ahn. Spatially-aware transformer for embodied agents.arXiv preprint arXiv:2402.15160, 2024. 3

    2024 arXiv

  16. [24]

    Local all-pair correspondence for point tracking

    Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon-Young Lee. Local all-pair correspondence for point tracking. InEuropean Conference on Computer Vision , pages 306–325. Springer, 2024. 5, 19

    2024

  17. [25]

    Simple and effective multi-paragraph reading comprehension

    Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehension. InACL, pages 845–855, 2018. 7

    2018

  18. [26]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 6, 19

    2017

  19. [27]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning , pages 933–941. PMLR, 2017. 5

    2017

  20. [28]

    Quar-vla: Vision-language-action model for quadruped robots

    Pengxiang Ding, Han Zhao, Wenjie Zhang, Wenxuan Song, Min Zhang, Siteng Huang, Ningxi Yang, and Donglin Wang. Quar-vla: Vision-language-action model for quadruped robots. InEuropean Conference on Computer Vision, pages 352–367. Springer, 2024. 1, 4, 6, 8, 10

    2024

  21. [29]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  22. [30]

    Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023. 3, 4

    2023 arXiv

  23. [31]

    Centernet: Keypoint triplets for object detection

    Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6569–6578, 2019. 2

    2019

  24. [32]

    Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions

    Cunxin Fan, Xiaosong Jia, Yihang Sun, Yixiao Wang, Jianglan Wei, Ziyang Gong, Xiangyu Zhao, Masayoshi Tomizuka, Xue Yang, Junchi Yan, et al. Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions. arXiv preprint arXiv:2505.02152 , 2025. 4

    2025

  25. [33]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. InCVPR, pages 19358–19369, 2023. 3

    2023

  26. [34]

    MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv: 2306.13394 , 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv: 2306.13394 , 2023. 7

    2023 arXiv

  27. [35]

    Rlafford: End-to-end affordance learning for robotic manipulation

    Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. Rlafford: End-to-end affordance learning for robotic manipulation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5880–5886. IEEE, 2023. 3

    2023

  28. [36]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InCVPR, pages 6904–6913, 2017. 3 12

    2017

  29. [37]

    Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

    Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InCVPR, pages 3608–3617,

  30. [38]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 15979–15988, 2022. 3

    2022

  31. [39]

    3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems , 36: 20482–20494, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems , 36: 20482–20494, 2023. 7

    2023

  32. [40]

    Dita: Scaling diffusion transformer for generalist vision-language-action policy.arXiv preprint arXiv:2503.19757, 2025

    Zhi Hou, Tianyi Zhang, Yuwen Xiong, Haonan Duan, Hengjun Pu, Ronglei Tong, Chengyang Zhao, Xizhou Zhu, Yu Qiao, Jifeng Dai, et al. Dita: Scaling diffusion transformer for generalist vision-language-action policy.arXiv preprint arXiv:2503.19757, 2025. 4

    2025 arXiv

  33. [41]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168 , 2023. 7, 9

    2023 arXiv

  34. [42]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers.CoRR, 2023

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers.CoRR, 2023. 7

    2023

  35. [43]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871 ,

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871 ,

    arXiv

  36. [44]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022. 4

    2022 arXiv

  37. [45]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. InConference on Robot Learning, pages 991–1002. PMLR, 2022. 1, 4

    2022

  38. [46]

    Robobrain: A unified brain model for robotic manipulation from abstract to concrete

    Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. arXiv preprint arXiv:2502.21257 , 2025. 1, 7, 8

    2025 arXiv

  39. [47]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–4916, 2021. 3

    2021

  40. [48]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InECCV, pages 235–251, 2016. 1, 7

    2016

  41. [49]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 1, 2, 4, 7, 9, 10

    2024 arXiv

  42. [50]

    Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692 , 2023

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692 , 2023. 3

    2023 arXiv

  43. [51]

    Cornernet: Detecting objects as paired keypoints

    Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. InProceedings of the European conference on computer vision (ECCV) , pages 734–750, 2018. 2

    2018

  44. [52]

    Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.The International journal of robotics research, 37(4-5):421–436, 2018

    Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.The International journal of robotics research, 37(4-5):421–436, 2018. 3

    2018

  45. [53]

    Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension.arXiv preprint arXiv:2404.16790 , 2024

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension.arXiv preprint arXiv:2404.16790 , 2024. 7 13

    2024 arXiv

  46. [54]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326 , 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326 , 2024. 7, 8

    2024 arXiv

  47. [55]

    Learning agile skills via adversarial imitation of rough partial demonstrations

    Chenhao Li, Marin Vlastelica, Sebastian Blaes, Jonas Frey, Felix Grimminger, and Georg Martius. Learning agile skills via adversarial imitation of rough partial demonstrations. InConference on Robot Learning, pages 342–352. PMLR, 2023. 5

    2023

  48. [56]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. InICML, pages 19730–19742, 2023. 3

    2023

  49. [57]

    Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text

    Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, et al. Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text. InThe Thirteenth International Conference on Learning Representations, 2025. 3

    2025

  50. [58]

    Exploring plain vision transformer backbones for object detection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. arXiv preprint arXiv:2203.16527 , 2022. 5

    2022 arXiv

  51. [59]

    Robotic visual instruction

    Yanbang Li, Ziyang Gong, Haoyang Li, Xiaoqi Huang, Haolan Kang, Guangping Bai, and Xianzheng Ma. Robotic visual instruction. arXiv preprint arXiv:2505.00693 , 2025. 4

    2025 arXiv

  52. [60]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 9493–9500. IEEE, 2023. 4

    2023

  53. [61]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 26689–26699, 2024. 3, 8

    2024

  54. [62]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. InECCV, pages 740–755, 2014. 3

    2014

  55. [63]

    Coarse correspondence elicit 3d spacetime understanding in multimodal language model

    Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondence elicit 3d spacetime understanding in multimodal language model. arXiv preprint arXiv:2408.00754, 2024. 3

    2024 arXiv

  56. [64]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 3

    2023

  57. [65]

    Mmbench: Is your multi-modal model an all-around player? arXiv: 2307.06281 , 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? arXiv: 2307.06281 , 2023. 7

    2023 arXiv

  58. [66]

    On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023

    Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023. 7

    2023 arXiv

  59. [67]

    Mminstruct: A high-quality multi-modal instruction tuning dataset with extensive diversity

    Yangzhou Liu, Yue Cao, Zhangwei Gao, Weiyun Wang, Zhe Chen, Wenhai Wang, Hao Tian, Lewei Lu, Xizhou Zhu, Tong Lu, et al. Mminstruct: A high-quality multi-modal instruction tuning dataset with extensive diversity. Science China Information Sciences , 67(12):1–16, 2024. 6, 19

    2024

  60. [68]

    Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language

    Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language. arXiv preprint arXiv:2305.05662 , 2023. 3

    2023 arXiv

  61. [69]

    Mono- internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono- internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. arXiv preprint arXiv:2410.08202 , 2024. 1, 3

    arXiv

  62. [70]

    Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474 , 2022

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474 , 2022. 1, 7

    2022 arXiv

  63. [71]

    Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems , 36:655–677, 2023

    Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent- Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing ...

    2023

  64. [72]

    Walk these ways: Tuning robot control for generalization with multiplicity of behavior

    Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. InConference on Robot Learning, pages 22–31. PMLR, 2023. 1, 5

    2023

  65. [73]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InCVPR, pages 3195–3204, 2019. 3

    2019

  66. [74]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InACL, pages 2263–2279, 2022. 1, 7

    2022

  67. [75]

    Infograph- icvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infograph- icvqa. In W ACV, pages 1697–1706, 2022. 7

    2022

  68. [76]

    Quadrupedgpt: Towards a versatile quadruped agent in open-ended worlds

    Yuting Mei, Ye Wang, Sipeng Zheng, and Qin Jin. Quadrupedgpt: Towards a versatile quadruped agent in open-ended worlds. arXiv preprint arXiv:2406.16578 , 2024. 2

    2024 arXiv

  69. [77]

    Embodiedgpt: Vision-language pre-training via embodied chain of thought.Advances in Neural Information Processing Systems, 36:25081–25094, 2023

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought.Advances in Neural Information Processing Systems, 36:25081–25094, 2023. 2, 3

    2023

  70. [78]

    R3m: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601 , 2022

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601 , 2022. 4

    2022 arXiv

  71. [79]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. https://cdn.openai.com/gpt-4o-system-card.pdf, 2024. 7, 8, 10

    2024

  72. [80]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824 , 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824 , 2023. 3

    2023 arXiv

  73. [81]

    Gpt4scene: Understand 3d scenes from videos with vision-language models.arXiv preprint arXiv:2501.01428 , 2025

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models.arXiv preprint arXiv:2501.01428 , 2025. 6, 7, 9, 19

    2025 arXiv

  74. [82]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. 3

    2018

  75. [83]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 3

    2019

  76. [84]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  77. [85]

    Real-world robot learning with masked visual pre-training

    Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. InConference on Robot Learning, pages 416–426. PMLR, 2023. 4

    2023

  78. [86]

    Cliport: What and where pathways for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894–906. PMLR, 2022. 1, 4

    2022

  79. [87]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. InCVPR, 2019. 1, 3, 7

    2019

  80. [88]

    Open-world object manipulation using pre-trained vision-language models

    Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905 , 2023. 1, 4

    2023 arXiv

  81. [89]

    Smart: Self-supervised multi-task pretraining with control transformers.arXiv preprint arXiv:2301.09816 , 2023

    Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. Smart: Self-supervised multi-task pretraining with control transformers.arXiv preprint arXiv:2301.09816 , 2023. 4

    2023 arXiv

  82. [90]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. 3

    2023 arXiv

  83. [91]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530 , 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530 , 2024. 1, 3, 7...

    2024 arXiv

  84. [92]

    Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020 , 2025

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503...

    2025 arXiv

  85. [93]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 3

    2023 arXiv

  86. [94]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv: 2307.09288 , 2023. 3

    2023 arXiv

  87. [95]

    The all-seeing project: Towards panoptic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. In ICLR, 2024. 3

    2024

  88. [96]

    Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression

    Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Chengmeng Li, Xiaoyu Liu, Yaxin Peng, Chaomin Shen, et al. Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression. arXiv preprint arXiv:2412.03293 , 2024. 7

    2024 arXiv

  89. [97]

    Grok-1.5 vision preview

    X.ai. Grok-1.5 vision preview. https://x.ai/blog/grok-1.5v, 2024. 7

    2024

  90. [98]

    Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171 , 2024

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171 , 2024. 1, 7, 8

    2024 arXiv

  91. [99]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision).arXiv: 2309.17421 , 9, 2023

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision).arXiv: 2309.17421 , 9, 2023. 1, 3

    2023 arXiv

  92. [100]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800 , 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800 , 2024. 7

    2024 arXiv

  93. [102]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    arXiv

  94. [103]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv: 2311.16502 , 2023

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv: 2311.16502 , 2023. 7

    2023 arXiv

  95. [104]

    Robotic control via embodied chain-of-thought reasoning

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv preprint arXiv:2407.08693 , 2024. 7

    2024 arXiv

  96. [105]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019. 5

    2019

  97. [106]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852,

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852,

    arXiv

  98. [107]

    Multi3drefer: Grounding text description to multiple 3d objects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15225–15236,

  99. [108]

    Llava-next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, 2024. 8

    2024

  100. [109]

    3dvg-transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2928–2937,

  101. [110]

    Chatvla: Unified multimodal understanding and robot control with vision-language- action model

    Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with vision-language- action model. arXiv preprint arXiv:2502.14420 , 2025. 1, 2, 4, 7

    2025 arXiv

  102. [111]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125 , 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125 , 2024. 3

    2024 arXiv

  103. [112]

    Target- driven visual navigation in indoor scenes using deep reinforcement learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target- driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 3357–3364. IEE...

    2017

  104. [113]

    Robot parkour learning

    Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. InConference on Robot Learning CoRL , 2023. 3 17 A More Implementation Details A.1 Training Details Our VeBrain is optimized in a fully superv...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.