REVIEW 4 major objections 5 minor 137 references
AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read AntiGrounding lets a pretrained vision-language model pick robot trajectories directly from rendered images.
desk verdict A plausible system paper with an honest limitations section, but the core claim—VLM scores rank trajectories well enough to drive MPC—is never directly tested, and the reported numbers have inconsistencies. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the trajectory-based structured VQA evaluation inside the MPC loop. Candidate end-effector trajectories are sampled under an annealed exploration bias, filtered by collision and target-interaction constraints, and then color-coded and rendered onto multi-view scene images. A VLM ensemble answers four weighted sub-questions—safety, task alignment, efficiency, and physical feasibility—plus a viewpoint-clarity question; viewpoint confidence weights suppress unreliable views, and the aggregated score ranks trajectories, with the top scorer selected by the argmax rule. The 'anti-grounding' move is that the robot action itself becomes the visual input to the VLM rather than being converted to a compressed intermediate representation.
What would settle it
Take a fixed scene, generate candidate trajectories whose true success or failure is known from execution, render them with the paper's multi-view protocol, and ask the VLM ensemble to rank them. If the top-ranked trajectory fails more often than a randomly chosen feasible trajectory, or if ranking accuracy on fine-grained geometric perturbations is at chance, the central claim is falsified.
Extended reading notes
Core claim
The central discovery is that reversing the instruction-grounding pipeline removes the information bottleneck. The paper argues that existing methods project a language instruction into compressed intermediate forms—symbolic skill sequences, voxel value maps, or keypoint constraints—and that this discards fine-grained spatial and physical detail the VLM has already encoded. AntiGrounding instead lifts candidate action trajectories into the VLM's native representation space by rendering them from robot-centric viewpoints and evaluating the augmented images with a structured visual-question-answering template. Scores from an ensemble of VLMs are aggregated across sub-questions, views weighted by viewpoint confidence, and agents, and the argmax trajectory is executed in a closed-loop MPC. The paper reports that this zero-shot procedure outperforms all three baseline families on the tested manipulation suite, and that the optional offline policy refinement further raises success rates by adjusting the evaluation template from past execution outcomes.
Load-bearing premise
The whole system rests on the VLM's rendered-image scores being good enough to rank trajectories that differ by only centimeter-level placement and degree-level orientation; if those scores do not track real physical feasibility, the argmax selection picks the wrong motion.
Editorial extensions
If this is right
- A single pretrained VLM can serve as the cost function for closed-loop trajectory optimization, so a new manipulation task requires only a language instruction rather than reward engineering or fine-tuning.
- Multi-view rendering with viewpoint confidence weighting lets the VLM infer 3D scene structure without a dedicated 3D encoder.
- The offline meta-VLM refinement loop lets the evaluation policy self-correct after failures, so long-horizon performance improves with accumulated experience.
- On the reported task suite, the method beats approaches built on symbolic skill sequences, voxel value maps, and keypoint constraints, with the largest gaps on tasks requiring precise alignment and multi-stage sequencing.
Reading between the lines
- The mechanism is not tied to a specific robot arm: any planner that can propose candidate trajectories and render them could adopt the same VLM-as-judge loop, potentially extending to mobile manipulation, navigation, or bimanual tasks.
- A direct stress test would measure how the VLM ensemble's trajectory ranking degrades as candidate differences shrink below a few centimeters or degrees; the paper's own error analysis suggests this metric-level discrimination is the fragile point.
- The offline refinement could become an online self-supervised loop that uses execution success flags as reward labels to tune template weights, a step the paper leaves for future work.
- Because evaluation is purely image-and-text based, the framework could combine with other search strategies beyond the spherical sampler to explore higher-dimensional or contact-rich trajectory spaces.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AntiGrounding, a framework that inverts conventional instruction-grounding pipelines by rendering candidate robot trajectories from multiple viewpoints and asking a pre-trained VLM ensemble to score them via structured visual question answering. The highest-scoring trajectory is executed in a closed-loop MPC, with an optional offline meta-VLM module that refines the evaluation template from prior execution logs. The authors report success rates on eight real-robot manipulation tasks, comparing against Code as Policies, Voxposer, and Rekep, and claim zero-shot synthesis of closed-loop trajectories for novel tasks. The paper also includes ablations, an offline policy-optimization study, and a system error breakdown.
Significance. If the central premise holds, the paper contributes an interesting alternative to intermediate-representation approaches: instead of compressing VLM knowledge into symbolic skills or value maps, it uses rendered trajectory overlays as a native interface to the VLM's spatial and semantic priors. The use of an ensemble of five pre-trained VLMs in a closed-loop MPC, without task-specific fine-tuning, is a distinctive and potentially useful design point. The paper also honestly documents its own limitations in Section 4.4 and Appendix A.8, including that VLM evaluation errors dominate and that fine-grained metric geometric constraints remain a challenge. However, the empirical support for the central premise is currently incomplete: the manuscript never directly measures whether the VLM scores in Eq. (3) are positively correlated with physical task success, and the reported numerical comparisons suffer from inconsistencies and lack of statistical rigor. The concept is promising, but the evidence as presented is not yet commensurate with the strength of the central claim.
major comments (4)
- [Section 3.3, Eqs. (2)-(4); Section 4.4; Appendix A.8] The load-bearing assumption is that the VLM scores S_{j,t} in Eq. (3) and the argmax in Eq. (4) reliably rank physically feasible trajectories. The paper never validates this correlation directly, instead reporting end-of-task success that couples VLM scoring with trajectory sampling, constraint filtering, and execution. Section 4.4 states that VLM evaluation errors dominate the failure breakdown, and Appendix A.8 concedes that "fine-grained, metric geometric constraints (e.g., sub-centimeter concentricity...) purely from 2D renderings remains a significant challenge for current VLMs." This is precisely the regime the method targets. I ask the authors to provide a direct scoring-accuracy measurement, for example by comparing VLM rankings against ground-truth or human-annotated trajectory quality on a held-out set of candidate trajectories, with per-task correlation coefficients or top-k selection accuracy. Without this, the central claim that anti-grounding enables reliable decision making remains asserted rather than demonstrated.
- [Table 1 vs. Table 3] The success numbers are internally inconsistent. Table 1 reports AntiGrounding success of 6/10 for Circular Tape Stacking, while Table 3 reports 55.24% for the "No Prior" condition on the same task; Waste Sorting is 7/10 in Table 1 but 68.49% in Table 3. Additionally, in Table 1, Rekep beats AntiGrounding on Water Pouring (6/10 vs. 5/10) despite AntiGrounding's higher aggregate score. The authors need to clarify whether these tables use different trial counts, different task configurations, or different evaluation criteria, and report the exact number of trials and the definition of success for each table. As written, the inconsistencies undermine the quantitative comparison.
- [Section 4.1, Table 1; Section 4.2, Table 2] The main comparison in Table 1 is based on 10 trials per task with no confidence intervals, error bars, or significance tests, and Table 2 reports ablation variants from a single run while only AntiGrounding is averaged over three runs. Given the high variance suggested by the AntiGrounding ablation standard deviations in Table 2 (e.g., 46.67±9.87 for Water Pouring), the differences between AntiGrounding and Rekep on individual tasks (e.g., 5/10 vs. 6/10 on Water Pouring) could easily be noise. The authors should provide per-task confidence intervals and a significance test (or at least additional repetitions) for the main comparison, and report multiple runs for the ablations.
- [Section 4.3 and Appendix A.6] The offline policy refinement is evaluated on the same tasks whose execution outcomes are used to tune the sub-question weights and templates, which risks overfitting and makes the "improvement" numbers in Table 3 partly circular. The paper should clarify whether the offline refinement was evaluated on held-out task instances or only on the same task set, and ideally report performance on tasks not used for refinement. In addition, Table 3 does not state the number of trials per entry or whether the same trial set was used for both conditions; these details are necessary to assess the reported improvements.
minor comments (5)
- [Section 1] The sentence "Extensive experiments in both simulation and real-world environments demonstrate that AntiGrounding significantly outperforms conventional methods... particularly in complex manipulation tasks in complex manipulation tasks encompassing precise spatial reasoning" contains a duplicated phrase that should be removed.
- [Figure 3 caption] The caption contains a typo: "Sim Buider" should be "Sim Builder."
- [Appendix A.5.1] The sentence "Each model processes the same pipline in parallel" contains a typo: "pipline" should be "pipeline."
- [Appendix A.2.3] The Water Pouring task is described as pouring from a "green cup" into a "stainless steel bowl," whereas Section 1 and Figure 7 refer to a teapot and cup; please reconcile the object descriptions across the paper.
- [Section 4.2] The ablation paragraph says AG-MVECW "led to unstable trajectory assessments" and AG-VGTG "severely affected trajectory quality," but the corresponding rows in Table 2 do not include error bars or significance tests; the text should be moderated to match the evidence presented.
Circularity Check
No significant circularity: the central zero-shot claim is externally evaluated, and the optional refinement module's in-domain tuning is a limitation, not a circular derivation.
full rationale
The paper's main derivation is an MPC loop in which pre-trained VLMs score externally rendered candidate trajectories (Eqs. 2-4), and success is measured by external task completion, not by the VLM's own scores. No parameter of the main method is fitted to the success labels that are then reported as the headline result: the VLM ensemble and hand-set template weights are fixed for the zero-shot comparisons in Table 1, and the baselines are independent systems. There is no load-bearing self-citation chain: the cited prior work on VLMs, spatial reasoning, Real2Sim, and MPC is external to this paper's authors. The strongest caveats are empirical, not circular: Section 4.4 reports that VLM evaluation errors dominate failures, and Appendix A.8 concedes that fine-grained metric geometric evaluation from 2D renderings remains a significant VLM challenge. Those concessions weaken the premise that VLM scores can rank centimeter-level trajectories, but they do not make the derivation reduce to its inputs. The optional offline policy refinement (Sec. 4.3, Appendix A.6) updates template weights from execution outcomes on the same task set and then reports improved success on those tasks; this is an in-domain fitting/adaptation evaluation rather than a held-out prediction, and it is not presented as the zero-shot claim. It is a methodological limitation, not a circular step under the strict definition used here.
Assumptions & free parameters
free parameters (5)
- Sub-question weights w_k =
safety 0.25, task alignment 0.35, efficiency 0.20, physical feasibility 0.20 (initial, later tuned by meta-VLM)
- Sampling radius and angle schedule =
R0=0.25 m, Rmin=0.1 m, lambda_R=0.693, theta0=90 deg, theta_min=30 deg, lambda_theta=0.712
- Viewpoint confidence decay lambda_C =
0.5
- Number of active views k =
3
- Prediction horizon H_t and candidate count N =
unspecified
assumptions (5)
- domain assumption VLM ensemble members return parseable, consistent numeric scores for the structured VQA template.
- domain assumption Rendering candidate trajectories as colored overlays on multi-view images preserves enough spatial and geometric information for the VLM to judge them.
- domain assumption The Real2Sim2Real pipeline (SPAR3D reconstruction, Scalable Real2Sim physical identification, SAM-6D pose alignment) is accurate enough that simulated evaluation transfers to the real robot.
- domain assumption Rigid coupling between the grasped object and the end-effector is maintained during manipulation.
- ad hoc to paper Prompt rule requiring significant score variance across trajectories does not distort rankings.
Cite this review
Pith. "Pith review of AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making." pith.science (2026). https://pith.science/paper/ZRISXQ45
@misc{pith2026250612374,
author = {Pith},
title = {Pith review of: AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZRISXQ45}},
note = {Machine review of arXiv:2506.12374}
}
read the original abstract
Vision-Language Models (VLMs) encode knowledge and reasoning capabilities for robotic manipulation within high-dimensional representation spaces. However, current approaches often project them into compressed intermediate representations, discarding important task-specific information such as fine-grained spatial or semantic details. To address this, we propose AntiGrounding, a new framework that reverses the instruction grounding process. It lifts candidate actions directly into the VLM representation space, renders trajectories from multiple views, and uses structured visual question answering for instruction-based decision making. This enables zero-shot synthesis of optimal closed-loop robot trajectories for new tasks. We also propose an offline policy refinement module that leverages past experience to enhance long-term performance. Experiments in both simulation and real-world environments show that our method outperforms baselines across diverse robotic manipulation tasks.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022
2022
-
[2]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, pages 8748–8763, 2021
2021
-
[3]
Scaling up visual and vision-language representation learning with noisy text supervision
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational Conference on Machine Learning, pages 4904–4916, 2021
2021
-
[4]
Zero-shot text-to-image generation
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational Conference on Machine Learning, pages 8821–8831. PMLR, 2021
2021
-
[5]
Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023
arXiv 2023
-
[6]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023
arXiv 2023
-
[7]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[8]
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022
arXiv 2022
Show all 137 references
-
[9]
Open-vocabulary queryable scene representations for real world planning
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages ...
2023
-
[10]
Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action
Dhruv Shah, Bła ˙zej Osi´nski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. InConference on robot learning, pages 492–504. PMLR, 2023
2023
-
[11]
So- cratic models: Composing zero-shot multimodal reasoning with language.arXiv preprint arXiv:2204.00598, 2022
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. So- cratic models: Composing zero-shot multimodal reasoning with language.arXiv preprint arXiv:2204.00598, 2022
2022 arXiv
-
[12]
Language models as zero-shot planners: Extracting actionable knowledge for embodied agents
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. InInternational conference on machine learning, pages 9118–9147. PMLR, 2022
2022
-
[13]
Code as policies: Language model programs for embodied control
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023. 10
2023
-
[14]
V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023
2023 arXiv
-
[15]
Pivot: Iterative visual prompting elicits actionable knowledge for vlms.arXiv preprint arXiv:2402.07872, 2024
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms.arXiv preprint arXiv:2402.07872, 2024
2024 arXiv
-
[16]
Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024
2024 arXiv
-
[17]
Robodexvlm: Visual language model-enabled task planning and motion control for dexterous robot manipu- lation.arXiv preprint arXiv:2503.01616, 2025
Haichao Liu, Sikai Guo, Pengfei Mai, Jiahang Cao, Haoang Li, and Jun Ma. Robodexvlm: Visual language model-enabled task planning and motion control for dexterous robot manipu- lation.arXiv preprint arXiv:2503.01616, 2025
2025 arXiv
-
[18]
Roboground: Robotic manipulation with grounded vision- language priors.arXiv preprint arXiv:2504.21530, 2025
Haifeng Huang, Xinyi Chen, Yilun Chen, Hao Li, Xiaoshen Han, Zehan Wang, Tai Wang, Jiangmiao Pang, and Zhou Zhao. Roboground: Robotic manipulation with grounded vision- language priors.arXiv preprint arXiv:2504.21530, 2025
2025 arXiv
-
[19]
Llm-grounder: Open-vocabulary 3d visual grounding with large lan- guage model as an agent
Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Llm-grounder: Open-vocabulary 3d visual grounding with large lan- guage model as an agent. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7...
2024
-
[20]
Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors.arXiv preprint arXiv:2403.13438, 2024
Chenyang Ma, Kai Lu, Ta-Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors.arXiv preprint arXiv:2403.13438, 2024
2024 arXiv
-
[21]
Spatialvlm: Endowing vision-language models with spatial reasoning capabilities
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024
2024
-
[22]
Spatialrgpt: Grounded spatial reasoning in vision language models.arXiv preprint arXiv:2406.01584, 2024
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models.arXiv preprint arXiv:2406.01584, 2024
2024 arXiv
-
[23]
Capture: Evaluating spatial reasoning in vision language models via occluded object counting.arXiv preprint arXiv:2504.15485, 2025
Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Capture: Evaluating spatial reasoning in vision language models via occluded object counting.arXiv preprint arXiv:2504.15485, 2025
2025 arXiv
-
[24]
Mind the gap: Benchmarking spatial reasoning in vision-language models.arXiv preprint arXiv:2503.19707, 2025
Ilias Stogiannidis, Steven McDonagh, and Sotirios A Tsaftaris. Mind the gap: Benchmarking spatial reasoning in vision-language models.arXiv preprint arXiv:2503.19707, 2025
2025 arXiv
-
[25]
Zero-shot visual reasoning by vision- language models: Benchmarking and analysis
Aishik Nagar, Shantanu Jaiswal, and Cheston Tan. Zero-shot visual reasoning by vision- language models: Benchmarking and analysis. In2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024
2024
-
[26]
How to enable llm with 3d capacity? a survey of spatial reasoning in llm.arXiv preprint arXiv:2504.05786, 2025
Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. How to enable llm with 3d capacity? a survey of spatial reasoning in llm.arXiv preprint arXiv:2504.05786, 2025
2025
-
[27]
Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024
2024 arXiv
-
[28]
Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024
2024 arXiv
-
[29]
Agent3d-zero: An agent for zero-shot 3d understanding
Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. InEuropean Conference on Computer Vision, pages 186–202. Springer, 2024
2024
-
[30]
Shapellm: Universal 3d object understanding for embodied interaction
Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In European Conference on Computer Vision, pages 214–238. Springer, 2024
2024
-
[31]
Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024. 11
2024 arXiv
-
[32]
Inf-llava: Dual-perspective perception for high-resolution multimodal large language model
Yiwei Ma, Zhibin Wang, Xiaoshuai Sun, Weihuang Lin, Qiang Zhou, Jiayi Ji, and Rongrong Ji. Inf-llava: Dual-perspective perception for high-resolution multimodal large language model. arXiv preprint arXiv:2407.16198, 2024
2024 arXiv
-
[33]
G-llava: Solving geometric problem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023
2023 arXiv
-
[34]
Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models
Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models. In2024 IEEE/RSJ International Conference on Intel...
2024
-
[35]
Robovqa: Multimodal long-horizon reasoning for robotics
Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakr- ishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In2024 IEEE International Conference on ...
2024
-
[36]
Mqa: Answering the question via robotic manipulation.arXiv preprint arXiv:2003.04641, 2020
Yuhong Deng, Di Guo, Xiaofeng Guo, Naifu Zhang, Huaping Liu, and Fuchun Sun. Mqa: Answering the question via robotic manipulation.arXiv preprint arXiv:2003.04641, 2020
2003 arXiv
-
[37]
Robotvqa—a scene-graph-and deep-learning-based visual question answering system for robot manipulation
Franklin Kenghagho Kenfack, Feroz Ahmed Siddiky, Ferenc Balint-Benczedi, and Michael Beetz. Robotvqa—a scene-graph-and deep-learning-based visual question answering system for robot manipulation. In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)...
2020
-
[38]
A visual questioning answering approach to enhance robot localization in indoor environments.Frontiers in Neurorobotics, 17:1290584, 2023
Juan Diego Peña-Narvaez, Francisco Martín, José Miguel Guerrero, and Rodrigo Pérez- Rodríguez. A visual questioning answering approach to enhance robot localization in indoor environments.Frontiers in Neurorobotics, 17:1290584, 2023
2023
-
[39]
Vlmpc: Vision-language model predictive control for robotic manipulation.arXiv preprint arXiv:2407.09829, 2024
Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, and Wei Zhang. Vlmpc: Vision-language model predictive control for robotic manipulation.arXiv preprint arXiv:2407.09829, 2024
2024 arXiv
-
[40]
Re- flective planning: Vision-language models for multi-stage long-horizon robotic manipulation
Yunhai Feng, Jiaming Han, Zhuoran Yang, Xiangyu Yue, Sergey Levine, and Jianlan Luo. Re- flective planning: Vision-language models for multi-stage long-horizon robotic manipulation. arXiv preprint arXiv:2502.16707, 2025
2025 arXiv
-
[41]
Open-world task and motion planning via vision-language model inferred constraints.arXiv preprint arXiv:2411.08253, 2024
Nishanth Kumar, William Shen, Fabio Ramos, Dieter Fox, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Caelan Reed Garrett. Open-world task and motion planning via vision-language model inferred constraints.arXiv preprint arXiv:2411.08253, 2024
2024
-
[42]
Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023
2023 arXiv
-
[43]
Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022
2022 arXiv
-
[44]
Robomamba: Multimodal state space model for efficient robot reasoning and manipulation.arXiv preprint arXiv:2406.04339, 2024
Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation.arXiv preprint arXiv:2406.04339, 2024
2024 arXiv
-
[45]
Scaling proprioceptive-visual learn- ing with heterogeneous pre-trained transformers.Advances in Neural Information Processing Systems, 37:124420–124450, 2024
Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visual learn- ing with heterogeneous pre-trained transformers.Advances in Neural Information Processing Systems, 37:124420–124450, 2024
2024
-
[46]
Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023
2023 arXiv
-
[47]
Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.arXiv preprint arXiv:2401.02117, 2024
Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.arXiv preprint arXiv:2401.02117, 2024
2024 arXiv
-
[48]
Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 12
2024 arXiv
-
[49]
Rdt-1b: a diffusion foundation model for bimanual manipulation
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024
2024 arXiv
-
[50]
Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression.arXiv preprint arXiv:2412.03293, 2024
Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Chengmeng Li, Xiaoyu Liu, Yaxin Peng, Chaomin Shen, et al. Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression.arXiv preprint arXiv:2412.03293, 2024
2024 arXiv
-
[51]
π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024
-
[52]
π0.5: A vision- language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5: A vision- language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025
2025 arXiv
-
[53]
Manipulate-anything: Automating real-world robots using vision-language models.arXiv preprint arXiv:2406.18915, 2024
Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-anything: Automating real-world robots using vision-language models.arXiv preprint arXiv:2406.18915, 2024
2024 arXiv
-
[54]
Skillman—a skill-based robotic manipulation framework based on perception and reasoning.Robotics and Autonomous Systems, 134:103653, 2020
Mohammed Diab, Mihai Pomarlan, Daniel Beßler, Aliakbar Akbari, Jan Rosell, John Bate- man, and Michael Beetz. Skillman—a skill-based robotic manipulation framework based on perception and reasoning.Robotics and Autonomous Systems, 134:103653, 2020
2020
-
[55]
A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint rewards.arXiv preprint arXiv:2502.08643, 2025
Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri, Li Fei-Fei, Svetlana Lazebnik, and Yunzhu Li. A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint rewards.arXiv preprint arXiv:2502.08643, 2025
2025 arXiv
-
[56]
Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation.arXiv preprint arXiv:2502.13143, 2025
Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, et al. Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation.arXiv preprint arXiv:2502.13143, 2025
2025
-
[57]
Geomanip: Geometric constraints as general interfaces for robot manipulation
Weiliang Tang, Jia-Hui Pan, Yun-Hui Liu, Masayoshi Tomizuka, Li Erran Li, Chi-Wing Fu, and Mingyu Ding. Geomanip: Geometric constraints as general interfaces for robot manipulation. arXiv preprint arXiv:2501.09783, 2025
2025 arXiv
-
[58]
Kuda: Keypoints to unify dynamics learning and visual prompting for open-vocabulary robotic manipulation.arXiv preprint arXiv:2503.10546, 2025
Zixian Liu, Mingtong Zhang, and Yunzhu Li. Kuda: Keypoints to unify dynamics learning and visual prompting for open-vocabulary robotic manipulation.arXiv preprint arXiv:2503.10546, 2025
2025 arXiv
-
[59]
Robogsim: A real2sim2real robotic gaussian splatting simulator
Xinhai Li, Jialin Li, Ziheng Zhang, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Kuo-Kun Tseng, and Ruiping Wang. Robogsim: A real2sim2real robotic gaussian splatting simulator. arXiv preprint arXiv:2411.11839, 2024
2024 arXiv
-
[60]
Rl-gsbridge: 3d gaussian splatting based real2sim2real method for robotic manipulation learning.arXiv preprint arXiv:2409.20291, 2024
Yuxuan Wu, Lei Pan, Wenhua Wu, Guangming Wang, Yanzi Miao, Fan Xu, and Hesheng Wang. Rl-gsbridge: 3d gaussian splatting based real2sim2real method for robotic manipulation learning.arXiv preprint arXiv:2409.20291, 2024
2024 arXiv
-
[61]
Discovery and deployment of emergent robot swarm behaviors via represen- tation learning and real2sim2real transfer.arXiv preprint arXiv:2502.15937, 2025
Connor Mattson, Varun Raveendra, Ricardo Vega, Cameron Nowzari, Daniel S Drew, and Daniel S Brown. Discovery and deployment of emergent robot swarm behaviors via represen- tation learning and real2sim2real transfer.arXiv preprint arXiv:2502.15937, 2025
2025 arXiv
-
[62]
Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.arXiv preprint arXiv:2403.03949, 2024
Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.arXiv preprint arXiv:2403.03949, 2024
2024 arXiv
-
[63]
Rl-vigen: A reinforcement learning benchmark for visual generalization.Advances in Neural Information Processing Systems, 36:6720–6747, 2023
Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. Rl-vigen: A reinforcement learning benchmark for visual generalization.Advances in Neural Information Processing Systems, 36:6720–6747, 2023
2023
-
[64]
Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021
2021 arXiv
-
[65]
Efficient real2sim2real of continuum robots using deep reinforcement learning with koopman operator.IEEE Transactions on Industrial Electronics, 2025
Guanglin Ji, Qian Gao, Yin Xiao, and Zhenglong Sun. Efficient real2sim2real of continuum robots using deep reinforcement learning with koopman operator.IEEE Transactions on Industrial Electronics, 2025. 13
2025
-
[66]
Real-time per- ception meets reactive motion generation.IEEE Robotics and Automation Letters, 3(3):1864– 1871, 2018
Daniel Kappler, Franziska Meier, Jan Issac, Jim Mainprice, Cristina Garcia Cifuentes, Manuel Wüthrich, Vincent Berenz, Stefan Schaal, Nathan Ratliff, and Jeannette Bohg. Real-time per- ception meets reactive motion generation.IEEE Robotics and Automation Letters, 3(3):1864– 1871, 2018
2018
-
[67]
You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022
Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022
2022 arXiv
-
[68]
One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. InProceedings of the IEEE/CVF conference on compu...
2024
-
[69]
Sparp: Fast 3d object reconstruction and pose estimation from sparse views
Chao Xu, Ang Li, Linghao Chen, Yulin Liu, Ruoxi Shi, Hao Su, and Minghua Liu. Sparp: Fast 3d object reconstruction and pose estimation from sparse views. InEuropean Conference on Computer Vision, pages 143–163. Springer, 2024
2024
-
[70]
Zero123++: a single image to consistent multi-view diffusion base model.arXiv preprint arXiv:2310.15110, 2023
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model.arXiv preprint arXiv:2310.15110, 2023
2023 arXiv
-
[71]
Zero-1-to-3: Zero-shot one image to 3d object
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3d object. InProceedings of the IEEE/CVF international conference on computer vision, pages 9298–9309, 2023
2023
-
[72]
Get3d: A generative model of high quality 3d textured shapes learned from images.Advances In Neural Information Processing Systems, 35:31841–31854, 2022
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images.Advances In Neural Information Processing Systems, 35:31841–31854, 2022
2022
-
[73]
A-sdf: Learning disentangled signed distance functions for articulated shape represen- tation
Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape represen- tation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001–13011, 2021
2021
-
[74]
Ditto: Building digital twins of articulated objects from interaction
Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5616–5626, 2022
2022
-
[75]
Structure from action: Learn- ing interactions for articulated object 3d structure discovery.arXiv preprint arXiv:2207.08997, 2022
Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Structure from action: Learn- ing interactions for articulated object 3d structure discovery.arXiv preprint arXiv:2207.08997, 2022
2022 arXiv
-
[76]
Urdformer: A pipeline for constructing articulated simulation environments from real-world images.arXiv preprint arXiv:2405.11656, 2024
Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo, Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images.arXiv preprint arXiv:2405.11656, 2024
2024 arXiv
-
[77]
Paris: Part-level reconstruction and motion analysis for articulated objects
Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva. Paris: Part-level reconstruction and motion analysis for articulated objects. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 352–363, 2023
2023
-
[78]
Cage: controllable articulation generation
Jiayi Liu, Hou In Ivan Tam, Ali Mahdavi-Amiri, and Manolis Savva. Cage: controllable articulation generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17880–17889, 2024
2024
-
[79]
Sam-6d: Segment anything model meets zero-shot 6d object pose estimation
Jiehong Lin, Lihua Liu, Dekun Lu, and Kui Jia. Sam-6d: Segment anything model meets zero-shot 6d object pose estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27906–27916, 2024
2024
-
[80]
Gigapose: Fast and robust novel object pose estimation via one correspondence
Van Nguyen Nguyen, Thibault Groueix, Mathieu Salzmann, and Vincent Lepetit. Gigapose: Fast and robust novel object pose estimation via one correspondence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9903–9913, 2024
2024
-
[81]
Any6d: Model-free 6d pose estimation of novel objects.arXiv preprint arXiv:2503.18673, 2025
Taeyeop Lee, Bowen Wen, Minjun Kang, Gyuree Kang, In So Kweon, and Kuk-Jin Yoon. Any6d: Model-free 6d pose estimation of novel objects.arXiv preprint arXiv:2503.18673, 2025. 14
2025 arXiv
-
[82]
Foundationpose: Unified 6d pose estimation and tracking of novel objects
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17868–17879, 2024
2024
-
[83]
Foundpose: Unseen object pose estimation with foundation features
Evin Pınar Örnek, Yann Labbé, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. Foundpose: Unseen object pose estimation with foundation features. In European Conference on Computer Vision, pages 163–182. Springer, 2024
2024
-
[84]
A comprehensive survey on point cloud registration.arXiv preprint arXiv:2103.02690, 2021
Xiaoshui Huang, Guofeng Mei, Jian Zhang, and Rana Abbas. A comprehensive survey on point cloud registration.arXiv preprint arXiv:2103.02690, 2021
2021 arXiv
-
[85]
Deep learning-based point cloud registration: A comprehensive survey and taxonomy
Yu-Xin Zhang, Jie Gui, Baosheng Yu, Xiaofeng Cong, Xin Gong, Wenbing Tao, and Dacheng Tao. Deep learning-based point cloud registration: A comprehensive survey and taxonomy. arXiv preprint arXiv:2404.13830, 2024
2024 arXiv
-
[86]
A tutorial review on point cloud registrations: principle, classification, comparison, and technology challenges.Mathematical Problems in Engineering, 2021(1):9953910, 2021
Leihui Li, Riwei Wang, and Xuping Zhang. A tutorial review on point cloud registrations: principle, classification, comparison, and technology challenges.Mathematical Problems in Engineering, 2021(1):9953910, 2021
2021
-
[87]
A comprehensive survey of visual slam algorithms.Robotics, 11(1):24, 2022
Andréa Macario Barros, Maugan Michel, Yoann Moline, Gwenolé Corre, and Frédérick Carrel. A comprehensive survey of visual slam algorithms.Robotics, 11(1):24, 2022
2022
-
[88]
How nerfs and 3d gaussian splatting are reshaping slam: a survey.arXiv preprint arXiv:2402.13255, 4:1, 2024
Fabio Tosi, Youmin Zhang, Ziren Gong, Erik Sandström, Stefano Mattoccia, Martin R Oswald, and Matteo Poggi. How nerfs and 3d gaussian splatting are reshaping slam: a survey.arXiv preprint arXiv:2402.13255, 4:1, 2024
2024 arXiv
-
[89]
A survey on active simultaneous localization and mapping: State of the art and new frontiers.IEEE Transactions on Robotics, 39(3):1686–1705, 2023
Julio A Placed, Jared Strader, Henry Carrillo, Nikolay Atanasov, Vadim Indelman, Luca Carlone, and José A Castellanos. A survey on active simultaneous localization and mapping: State of the art and new frontiers.IEEE Transactions on Robotics, 39(3):1686–1705, 2023
2023
-
[90]
Scalable real2sim: Physics-aware asset generation via robotic pick-and-place setups.arXiv preprint arXiv:2503.00370, 2025
Nicholas Pfaff, Evelyn Fu, Jeremy Binagia, Phillip Isola, and Russ Tedrake. Scalable real2sim: Physics-aware asset generation via robotic pick-and-place setups.arXiv preprint arXiv:2503.00370, 2025
2025 arXiv
-
[91]
Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos
Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. arXiv preprint arXiv:2503.17973, 2025
2025 arXiv
-
[92]
Sim2real 2: Actively building explicit physics model for precise articulated object manipulation
Liqian Ma, Jiaojiao Meng, Shuntao Liu, Weihang Chen, Jing Xu, and Rui Chen. Sim2real 2: Actively building explicit physics model for precise articulated object manipulation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11698–11704. IEEE, 2023
2023
-
[93]
A real2sim2real method for robust object grasping with neural surface reconstruction
Luobin Wang, Runlin Guo, Quan Vuong, Yuzhe Qin, Hao Su, and Henrik Christensen. A real2sim2real method for robust object grasping with neural surface reconstruction. In2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pages 1–8. IEEE, 2023
2023
-
[94]
Spar3d: Stable point-aware reconstruction of 3d objects from single images.arXiv preprint arXiv:2501.04689, 2025
Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M Rehg, and Varun Jampani. Spar3d: Stable point-aware reconstruction of 3d objects from single images.arXiv preprint arXiv:2501.04689, 2025
2025 arXiv
-
[95]
Pointllm: Empowering large language models to understand point clouds
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. InEuropean Conference on Computer Vision, pages 131–147. Springer, 2024
2024
-
[96]
Point-nerf: Point-based neural radiance fields
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5438–5448, 2022
2022
-
[97]
Pointclip: Point cloud understanding by clip
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8552–8562, 2022
2022
-
[98]
Text2nerf: Text-driven 3d scene generation with neural radiance fields.IEEE Transactions on Visualization and Computer Graphics, 30(12):7749–7762, 2024
Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields.IEEE Transactions on Visualization and Computer Graphics, 30(12):7749–7762, 2024
2024
-
[99]
Pointr: Diverse point cloud completion with geometry-aware transformers
Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 12498–12507, 2021. 15
2021
-
[100]
V oxel set transformer: A set-to-set approach to 3d object detection from point clouds
Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang. V oxel set transformer: A set-to-set approach to 3d object detection from point clouds. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8417–8427, 2022
2022
-
[101]
3d-vla: A 3d vision-language-action generative world model.arXiv preprint arXiv:2403.09631, 2024
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model.arXiv preprint arXiv:2403.09631, 2024
2024 arXiv
-
[102]
An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023
2023 arXiv
-
[103]
Occllama: An occupancy-language-action generative world model for autonomous driving
Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Occllama: An occupancy-language-action generative world model for autonomous driving. arXiv preprint arXiv:2409.03272, 2024
2024 arXiv
-
[104]
Pointvla: Injecting the 3d world into vision-language-action models.arXiv preprint arXiv:2503.07511, 2025
Chengmeng Li, Junjie Wen, Yan Peng, Yaxin Peng, Feifei Feng, and Yichen Zhu. Pointvla: Injecting the 3d world into vision-language-action models.arXiv preprint arXiv:2503.07511, 2025
2025 arXiv
-
[105]
Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. arXiv preprint arXiv:2411.16537, 2024
2024
-
[106]
Unconditional scene graph generation
Sarthak Garg, Helisa Dhamo, Azade Farshad, Sabrina Musatian, Nassir Navab, and Federico Tombari. Unconditional scene graph generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 16362–16371, 2021
2021
-
[107]
Incorporating structured representations into pretrained vision & language models using scene graphs.arXiv preprint arXiv:2305.06343, 2023
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Dar- rell, and Amir Globerson. Incorporating structured representations into pretrained vision & language models using scene graphs.arXiv preprint arXiv:2305.06343, 2023
2023 arXiv
-
[108]
Metricprompt: Prompting model as a relevance metric for few-shot text classification
Hongyuan Dong, Weinan Zhang, and Wanxiang Che. Metricprompt: Prompting model as a relevance metric for few-shot text classification. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 426–436, 2023
2023
-
[109]
Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models.arXiv preprint arXiv:2409.09788, 2024
Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models.arXiv preprint arXiv:2409.09788, 2024
2024 arXiv
-
[110]
Affordancenet: An end-to-end deep learning approach for object affordance detection
Thanh-Toan Do, Anh Nguyen, and Ian Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In2018 IEEE international conference on robotics and automation (ICRA), pages 5882–5889. IEEE, 2018
2018
-
[111]
Visual and linguistic cues to graspable objects.Experimental brain research, 229:545–559, 2013
Andriy Myachykov, Rob Ellis, Angelo Cangelosi, and Martin H Fischer. Visual and linguistic cues to graspable objects.Experimental brain research, 229:545–559, 2013
2013
-
[112]
Integrating with multimodal information for enhancing robotic grasping with vision-language models.IEEE Transactions on Automation Science and Engineering, 2025
Zhou Zhao, Dongyuan Zheng, Yizi Chen, Jing Luo, Yanjun Wang, Panfeng Huang, and Chenguang Yang. Integrating with multimodal information for enhancing robotic grasping with vision-language models.IEEE Transactions on Automation Science and Engineering, 2025
2025
-
[113]
Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation
Mohak Bhardwaj, Balakumar Sundaralingam, Arsalan Mousavian, Nathan D Ratliff, Dieter Fox, Fabio Ramos, and Byron Boots. Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation. InConference on Robot Learning, pages 750–759. PMLR, 2022
2022
-
[114]
Real-time model predictive control for industrial manipulators with singularity-tolerant hierarchical task control
Jaemin Lee, Mingyo Seo, Andrew Bylard, Robert Sun, and Luis Sentis. Real-time model predictive control for industrial manipulators with singularity-tolerant hierarchical task control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12282– 12288. I...
2023
-
[115]
Kinematic-model-free predictive control for robotic manipulator target reaching with obstacle avoidance.Frontiers in Robotics and AI, 9:809114, 2022
Ahmad AlAttar, Digby Chappell, and Petar Kormushev. Kinematic-model-free predictive control for robotic manipulator target reaching with obstacle avoidance.Frontiers in Robotics and AI, 9:809114, 2022
2022
-
[116]
Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.arXiv preprint arXiv:1812.00568, 2018
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.arXiv preprint arXiv:1812.00568, 2018. 16
2018 arXiv
-
[117]
Deep visual foresight for planning robot motion
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. InIEEE International Conference on Robotics and Automation, pages 2786–2793, 2017
2017
-
[118]
Learning 3d dynamic scene represen- tations for robot manipulation.arXiv preprint arXiv:2011.01968, 2020
Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene represen- tations for robot manipulation.arXiv preprint arXiv:2011.01968, 2020
2011 arXiv
-
[119]
Object-centric forward modeling for model predictive control
Yufei Ye, Dhiraj Gandhi, Abhinav Gupta, and Shubham Tulsiani. Object-centric forward modeling for model predictive control. InConference on Robot Learning, pages 100–109. PMLR, 2020
2020
-
[120]
Learning language- conditioned robot behavior from offline data and crowd-sourced annotation
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. InConference on Robot Learning, pages 1303–1315, 2022
2022
-
[121]
A control-centric benchmark for video prediction
Stephen Tian, Chelsea Finn, and Jiajun Wu. A control-centric benchmark for video prediction. InInternational Conference on Learning Representations, 2022
2022
-
[122]
Sequential manipulation of deformable linear object networks with endpoint pose measurements using adaptive model predictive control
Tyler Toner, Vahidreza Molazadeh, Miguel Saez, Dawn M Tilbury, and Kira Barton. Sequential manipulation of deformable linear object networks with endpoint pose measurements using adaptive model predictive control. In2024 IEEE International Conference on Robotics and Automation...
2024
-
[123]
Obstacle avoidance shape control of deformable lin- ear objects with online parameters adaptation based on differentiable simulation.ROBOMECH Journal, 11(1):15, 2024
Changjian Ying and Kimitoshi Yamazaki. Obstacle avoidance shape control of deformable lin- ear objects with online parameters adaptation based on differentiable simulation.ROBOMECH Journal, 11(1):15, 2024
2024
-
[124]
Bimanual deformable bag manipulation using a structure-of-interest based latent dynamics model.arXiv e-prints, pages arXiv–2401, 2024
Peng Zhou, Pai Zheng, Jiaming Qi, Chenxi Li, Chenguang Yang, David Navarro-Alarcon, and Jia Pan. Bimanual deformable bag manipulation using a structure-of-interest based latent dynamics model.arXiv e-prints, pages arXiv–2401, 2024
2024
-
[125]
Semantically informed mpc for context-aware robot exploration
Yash Goel, Narunas Vaskevicius, Luigi Palmieri, Nived Chebrolu, Kai O Arras, and Cyrill Stachniss. Semantically informed mpc for context-aware robot exploration. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11218–11225. IEEE, 2023
2023
-
[126]
Closing the perception-action loop for semantically safe navigation in semi-static environments
Jingxing Qian, Siqi Zhou, Nicholas Jianrui Ren, Veronica Chatrath, and Angela P Schoellig. Closing the perception-action loop for semantically safe navigation in semi-static environments. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 11641– 1164...
2024
-
[127]
Automatic configuration of multi-agent model predictive controllers based on semantic graph world models
Koen de V os, Elena Torta, Herman Bruyninckx, CA López Martínez, and MJG van de Molengraft. Automatic configuration of multi-agent model predictive controllers based on semantic graph world models. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7...
2024
-
[128]
Optimization by simulated annealing
Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. Optimization by simulated annealing. science, 220(4598):671–680, 1983
1983
-
[129]
Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023
2023 arXiv
-
[130]
Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692, 2024
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692, 2024
2024 arXiv
-
[131]
Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding.arXiv preprint arXiv:2412.04383, 2024
Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang. Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding.arXiv preprint arXiv:2412.04383, 2024
2024 arXiv
-
[132]
3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023
2023
-
[133]
Is a picture worth a thousand words? delving into spatial reasoning for vision language models
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392–75421, 2024
2024
-
[134]
On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021. 17
2021 arXiv
-
[135]
Eyes wide shut? exploring the visual shortcomings of multimodal llms
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024
2024
-
[136]
Beyond semantics: Rediscovering spatial awareness in vision-language models.arXiv preprint arXiv:2503.17349, 2025
Jianing Qi, Jiawei Liu, Hao Tang, and Zhigang Zhu. Beyond semantics: Rediscovering spatial awareness in vision-language models.arXiv preprint arXiv:2503.17349, 2025
2025
-
[137]
You are a helpful UR5e robot arm t r a j e c t o r y analyzer
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. 18 A Technical Appendices and Supplementary Material A.1...
2023 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.