REVIEW 3 major objections 5 minor 1 cited by
Foundation Model Driven Robotics: A Comprehensive Review
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This review argues that foundation models now act as robots' cognitive layer—open-vocabulary perception, task reasoning, code-generated control—while speed, grounding, data, safety, and interpretability still block deployment.
desk verdict A serviceable survey with a helpful taxonomy, undercut by an overclaimed 'comprehensive/critical' framing and at least one citation misattribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery that carries the argument is a four-part taxonomy of integration strategies—simulation-driven design, open-world execution, sim-to-real transfer, and adaptable robotics—overlaid on the classical perception-planning-control-interaction decomposition. Within that frame, the paper highlights the concrete mechanisms that make foundation models act as cognitive layers: SayCan-style grounding, where the language model proposes actions and an affordance model filters them by feasibility; inner-monologue closed-loop replanning, where the model revises its plan from textual feedback about what the robot sees; code-as-policies, where the model writes executable robot programs from natural language; and vision-language-action models such as RT-2 and GR00T that map multimodal inputs directly to motor commands. Each mechanism is presented as a component that can be combined with others in an integrated system.
What would settle it
Re-run the key cited systems (SayCan, RT-2, Code as Policies) on a fixed benchmark with novel objects, unseen layouts, and varied lighting while keeping prompts and weights unchanged; if zero-shot success rates fall to near zero or fail to beat a scripted baseline, the review's central claim of zero-shot generalization would be refuted.
Extended reading notes
Core claim
The central claim is that LLMs and VLMs, though trained on internet text and images rather than robot experience, can be integrated into every subsystem of a robot—perception, planning, control, and interaction—to produce capabilities that classical pipelines lack: naming and finding objects never seen in training, decomposing novel instructions into actionable steps, writing control code on the fly, and conversing with people in natural language. The paper argues that the key to making these capabilities reliable is coupling the model's semantic knowledge with grounding mechanisms: affordance checks that veto infeasible actions, closed-loop feedback that lets the model see the results of its plan, modular architectures that separate reasoning from physics prediction from motor control, and simulation pipelines that generate training data and test scenarios. Read sympathetically, the review claims the field is converging on a system-level recipe—language and vision priors plus embodiment-aware filtering—that is the practical route to open-world, general-purpose robots.
Load-bearing premise
The review's conclusions rest on the assumption that the papers it chose to discuss are a fair sample of the field and that their reported results are accurate, since no systematic search or quantitative synthesis is given.
Editorial extensions
If this is right
- Robots can be instructed in natural language for tasks they were not trained on, provided the model's proposals are filtered by physical feasibility checks.
- Closed-loop feedback (inner monologue) substantially improves long-horizon task success compared to one-shot planning.
- Simulation can be automated: language models generate environments, rewards, and domain-randomization curricula, reducing manual engineering for sim-to-real transfer.
- On-the-fly code generation gives robot control an inspectable, debuggable form, at the price of new safety risks from arbitrary code execution.
- The main obstacles to deployment are not conceptual but practical: inference latency and compute, lack of robot-grounded data, missing modalities like touch and proprioception, and unguaranteed safety.
Reading between the lines
- If the review's framing is right, the field's next validation step should be a shared benchmark that forces the same foundation-model stack to work across multiple embodiments and novel scenes, since the current evidence is a patchwork of system-specific demos.
- The paper's own symptom of mis-citing GenManip as RT-2 suggests that survey-level claims about which system does what may drift; readers should verify primary sources before building on secondary descriptions.
- The identified bottlenecks point toward a testable prediction: systems that pair a frozen LLM or VLM with a fast, lightweight verifier (an affordance filter or control-theoretic validator) will outpace monolithic end-to-end VLA models in real-world deployment.
- The review's four-part taxonomy could be used as a diagnostic checklist for whether a robot system is genuinely foundation-model-driven or merely uses an LLM as an isolated component.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper is a narrative survey of foundation model applications in robotics, organized around four themes: simulation-driven robotics, open-world execution, sim-to-real transfer, and adaptable robotic systems. It reviews how LLMs and VLMs contribute to perception, planning, control, and human-robot interaction, and it closes with a list of limitations and future research directions. The central claim is that foundation models have introduced a transformative paradigm in robotics, enabling open-vocabulary perception, high-level reasoning, and code generation for robot behaviors, with remaining bottlenecks in real-time operation, data grounding, safety, and interpretability.
Significance. If the survey's synthesis is reliable, it provides a useful and reasonably up-to-date organization of a fast-moving field, and its emphasis on integrated, system-level strategies distinguishes it from surveys that treat perception, planning, and control in isolation. The paper's main strengths are its broad coverage, its four-part taxonomy, and its explicit acknowledgment of practical bottlenecks such as real-time latency, embodiment gaps, and safety. There are no derivations or fitted parameters, so the survey does not suffer from the circularity that can affect benchmark-driven papers; the authors' own prior work appears only as incidental examples. However, the paper's value as a 'critical' and 'comprehensive' review is limited by the absence of a systematic methodology and by the uncritical reproduction of selected success rates from primary sources.
major comments (3)
- [Abstract; Section I; Section II.B] The paper calls itself a 'comprehensive' and 'critical' review, but it provides no methodological section and no search strategy, inclusion criteria, quality assessment, or data-extraction protocol. The 175 cited references are therefore presented as a convenience sample, and the central synthesis in Section II.B ('enabled unprecedented capabilities... indicating their potential for zero-shot generalization') inherits the uncertainty of that unrepresentative evidence base. Please add a methods subsection that defines how papers were selected, how they were assessed for quality, and how reported results were verified.
- [Section III.C] The statement 'GenManip [108]' is a citation error: reference [108] is Brohan et al.'s RT-2 paper, while GenManip is Gao et al., reference [130]. Because the survey's usefulness depends on accurate secondary reporting, this is a load-bearing reliability issue rather than a cosmetic one; please correct the citation and audit the remaining citation-to-claim pairings for similar mismatches.
- [Section VI.A and Section VI.B] Quantitative claims are reported without critical scrutiny. The text describes Zahedifar et al. as achieving '100% success' in adapting a two-link manipulator and Ouyang et al. as achieving 'over 70% success rates in simulation,' but it provides no information about the number of trials, task difficulty, baselines, or failure conditions. A review that promises a critical assessment should either interrogate these numbers or qualify them as authors' self-reported results rather than independently verified evidence.
minor comments (5)
- [Section III.C] There is an internal spacing typo in 'U AV agents'; it should read 'UAV agents'.
- [Section V.C] The claim that 'model-based validation methods offer a deeper understanding' is vague; the paragraph does not explain how the cited control-theoretic metrics in reference [166] are defined or why they are preferable to standard error metrics.
- [Section II.B and Section VII] The five limitations listed in Section VII closely repeat the limitations already stated in Section II.B; consider consolidating the two discussions to avoid redundancy and to preserve the forward-looking role of Section VII.
- [Table I] References [3] and [6] for GPT-3 and GPT-4 are secondary sources (a commentary and a news article) rather than the original technical reports; citing the primary model papers would give readers a more direct path to the source material.
- [Figures 2, 5, and 6] Several system-diagram figures do not indicate whether they are original illustrations or adapted from prior work; adding a credit note in each caption would clarify provenance.
Circularity Check
No circularity: the review's synthesis rests on external cited results, and the three author self-citations are incidental examples rather than load-bearing premises.
full rationale
This paper is a narrative survey, not a derivation or prediction pipeline, so the circularity patterns that involve fitted inputs, uniqueness theorems, or equations reducing to their own inputs do not apply. The central claim in Section II.B, that foundation models 'have enabled unprecedented capabilities, including open-vocabulary visual perception, high-level task reasoning, and code generation for robot behaviors,' is supported by independently reported external systems such as CLIP, PaLM-E, RT-1/RT-2, Code as Policies, and numerous other cited works. The three author self-citations are incidental: [23] is grouped with examples of instruction-following capabilities, [29] appears in a list of fine-tunable open-source models, and [166] supports a passing remark about control-theoretic metrics for quantifying sim-to-real divergence. None of these self-citations carries the paper's main argument, and no conclusion depends on accepting a self-cited result in place of independent evidence. There is no uniqueness theorem imported from the authors' prior work, no ansatz smuggled in by self-citation, and no known result merely renamed as a new contribution. The misattribution of GenManip to reference [108] instead of [130] in Section III.C is a factual accuracy issue in secondary reporting, not a circular step; similarly, the absence of a systematic search protocol concerns evidence representativeness and review methodology, not circularity. Because no step of the paper's synthesis is equivalent by construction to its inputs, the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The cited primary studies are reported accurately and their results are representative of the field.
- domain assumption Foundation models for robotics can be usefully equated with LLMs and VLMs, with other modalities treated as extensions.
- domain assumption The four-bucket organization (simulation-driven, open-world, sim-to-real, adaptable) is a meaningful partition of the literature.
Cite this review
Pith. "Pith review of Foundation Model Driven Robotics: A Comprehensive Review." pith.science (2026). https://pith.science/paper/PTS7H7W6
@misc{pith2026250710087,
author = {Pith},
title = {Pith review of: Foundation Model Driven Robotics: A Comprehensive Review},
year = {2026},
howpublished = {\url{https://pith.science/paper/PTS7H7W6}},
note = {Machine review of arXiv:2507.10087}
}
read the original abstract
The rapid emergence of foundation models, particularly Large Language Models (LLMs) and Vision-Language Models (VLMs), has introduced a transformative paradigm in robotics. These models offer powerful capabilities in semantic understanding, high-level reasoning, and cross-modal generalization, enabling significant advances in perception, planning, control, and human-robot interaction. This critical review provides a structured synthesis of recent developments, categorizing applications across simulation-driven design, open-world execution, sim-to-real transfer, and adaptable robotics. Unlike existing surveys that emphasize isolated capabilities, this work highlights integrated, system-level strategies and evaluates their practical feasibility in real-world environments. Key enabling trends such as procedural scene generation, policy generalization, and multimodal reasoning are discussed alongside core bottlenecks, including limited embodiment, lack of multimodal data, safety risks, and computational constraints. Through this lens, this paper identifies both the architectural strengths and critical limitations of foundation model-based robotics, highlighting open challenges in real-time operation, grounding, resilience, and trust. The review concludes with a roadmap for future research aimed at bridging semantic reasoning and physical intelligence through more robust, interpretable, and embodied models.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Language Movement Primitives: Grounding Language Models in Robot Motion
LMP lets a vision-language model set DMP weights and goals from a language command, achieving zero-demonstration tabletop manipulation with reported 80% success on 20 tasks.
Reference graph
Works this paper leans on
-
[108]
Rt-2: Vision- language-action models transfer web knowledge to robotic control,
A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choro- manski, T. Ding, D. Driess, A. Dubey, C. Finn et al. , “Rt-2: Vision- language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023
arXiv 2023
-
[130]
Genmanip: Llm-driven simulation for generalizable instruction-following manipulation,
N. Gao, Y . Chen, S. Yang, X. Chen, Y . Tian, H. Li, H. Huang, H. Wang, T. Wang, and J. Pang, “Genmanip: Llm-driven simulation for generalizable instruction-following manipulation,” in Proceedings KHAN et al.: FOUNDATION MODEL DRIVEN ROBOTICS: A COMPREHENSIVE REVIEW 16 of the Computer Vision and Pattern Recognition Conference, 2025, pp. 12 187–12 198
2025
-
[1]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017. KHAN et al.: FOUNDATION MODEL DRIVEN ROBOTICS: A COMPREHENSIVE REVIEW 13
2017
-
[2]
A survey on integration of large language models with intelligent robots,
Y . Kim, D. Kim, J. Choi, J. Park, N. Oh, and D. Park, “A survey on integration of large language models with intelligent robots,” Intelligent Service Robotics, vol. 17, no. 5, pp. 1091–1107, 2024
2024
-
[3]
Gpt-3: Its nature, scope, limits, and consequences,
L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” Minds and Machines , vol. 30, pp. 681–694, 2020
2020
-
[4]
Pathways language model (palm): Scaling to 540 billion parameters for breakthrough performance,
S. Narang and A. Chowdhery, “Pathways language model (palm): Scaling to 540 billion parameters for breakthrough performance,” Google AI Blog , 2022
2022
-
[5]
Language mod- els are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020
1901
-
[6]
Gpt-4 is here: what scientists think,
K. Sanderson, “Gpt-4 is here: what scientists think,” Nature, vol. 615, no. 7954, p. 773, 2023
2023
Show all 175 references
-
[7]
Swarm robotics: Past, present, and future [point of view],
M. Dorigo, G. Theraulaz, and V . Trianni, “Swarm robotics: Past, present, and future [point of view],” Proceedings of the IEEE , vol. 109, no. 7, pp. 1152–1165, 2021
2021
-
[8]
Deep learning in robotics: Survey on model structures and training strategies,
A. I. K ´aroly, P. Galambos, J. Kuti, and I. J. Rudas, “Deep learning in robotics: Survey on model structures and training strategies,” IEEE Transactions on Systems, Man, and Cybernetics: Systems , vol. 51, no. 1, pp. 266–279, 2020
2020
-
[9]
The strategic impacts of intelligent automation for knowledge and service work: An interdisciplinary review,
C. Coombs, D. Hislop, S. K. Taneva, and S. Barnard, “The strategic impacts of intelligent automation for knowledge and service work: An interdisciplinary review,”The Journal of Strategic Information Systems, vol. 29, no. 4, p. 101600, 2020
2020
-
[10]
Do as i can, not as i say: Grounding language in robotic affordances,
M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al., “Do as i can, not as i say: Grounding language in robotic affordances,”arXiv preprint arXiv:2204.01691, 2022
2022 arXiv
-
[11]
Lever- aging pre-trained large language models to construct and utilize world models for model-based task planning,
L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati, “Lever- aging pre-trained large language models to construct and utilize world models for model-based task planning,” Advances in Neural Informa- tion Processing Systems , vol. 36, pp. 79 081–79 094, 2023
2023
-
[12]
Code as policies: Language model programs for em- bodied control,
J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for em- bodied control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9493–9500
2023
-
[13]
Large language models for robotics: Opportunities, challenges, and perspectives,
J. Wang, E. Shi, H. Hu, C. Ma, Y . Liu, X. Wang, Y . Yao, X. Liu, B. Ge, and S. Zhang, “Large language models for robotics: Opportunities, challenges, and perspectives,” Journal of Automation and Intelligence , 2024
2024
-
[14]
Embodiedgpt: Vision-language pre-training via embodied chain of thought,
Y . Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y . Qiao, and P. Luo, “Embodiedgpt: Vision-language pre-training via embodied chain of thought,” Advances in Neural Information Processing Systems, vol. 36, pp. 25 081–25 094, 2023
2023
-
[15]
Enhancing robotic manipulation with ai feedback from multimodal large language models,
J. Liu, Y . Yuan, J. Hao, F. Ni, L. Fu, Y . Chen, and Y . Zheng, “Enhancing robotic manipulation with ai feedback from multimodal large language models,” arXiv preprint arXiv:2402.14245 , 2024
2024 arXiv
-
[16]
Integrating action knowledge and llms for task planning and situation handling in open worlds,
Y . Ding, X. Zhang, S. Amiri, N. Cao, H. Yang, A. Kaminski, C. Es- selink, and S. Zhang, “Integrating action knowledge and llms for task planning and situation handling in open worlds,” Autonomous Robots, vol. 47, no. 8, pp. 981–997, 2023
2023
-
[17]
Large language models for robotics: A survey,
F. Zeng, W. Gan, Y . Wang, N. Liu, and P. S. Yu, “Large language models for robotics: A survey,”arXiv preprint arXiv:2311.07226, 2023
2023
-
[18]
Large language models for multi- robot systems: A survey,
P. Li, Z. An, S. Abrar, and L. Zhou, “Large language models for multi- robot systems: A survey,” arXiv preprint arXiv:2502.03814 , 2025
2025 arXiv
-
[19]
Inner monologue: Embodied reasoning through planning with language models,
W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar et al. , “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608, 2022
2022 arXiv
-
[20]
Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,
W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in International conference on machine learning . PMLR, 2022, pp. 9118–9147
2022
-
[21]
Llm-planner: Few-shot grounded planning for embodied agents with large language models,
C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y . Su, “Llm-planner: Few-shot grounded planning for embodied agents with large language models,” in Proceedings of the IEEE/CVF international conference on computer vision , 2023, pp. 2998–3009
2023
-
[22]
Emergent abilities of large language models,
J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler et al. , “Emergent abilities of large language models,” arXiv preprint arXiv:2206.07682 , 2022
2022 arXiv
-
[23]
Leveraging vision-language models for manufacturing feature recognition in cad designs,
M. T. Khan, L. Chen, Y . H. Ng, W. Feng, N. Y . J. Tan, and S. K. Moon, “Leveraging vision-language models for manufacturing feature recognition in cad designs,” arXiv preprint arXiv:2411.02810 , 2024
2024 arXiv
-
[24]
Llm-based code generation method for golang compiler testing,
Q. Gu, “Llm-based code generation method for golang compiler testing,” in Proceedings of the 31st ACM Joint European Software En- gineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 2201–2203
2023
-
[25]
Nl2plan: Robust llm- driven planning from minimal text descriptions,
E. Gestrin, M. Kuhlmann, and J. Seipp, “Nl2plan: Robust llm- driven planning from minimal text descriptions,” arXiv preprint arXiv:2405.04215, 2024
2024
-
[26]
When robots get chatty: Grounding multimodal human-robot conversation and collaboration,
P. Allgeuer, H. Ali, and S. Wermter, “When robots get chatty: Grounding multimodal human-robot conversation and collaboration,” in International Conference on Artificial Neural Networks . Springer, 2024, pp. 306–321
2024
-
[27]
Understanding large-language model (llm)-powered human-robot interaction,
C. Y . Kim, C. P. Lee, and B. Mutlu, “Understanding large-language model (llm)-powered human-robot interaction,” in Proceedings of the 2024 ACM/IEEE international conference on human-robot interaction , 2024, pp. 371–380
2024
-
[28]
Llm-brain: Ai-driven fast generation of robot behaviour tree based on large language model,
A. Lykov and D. Tsetserukou, “Llm-brain: Ai-driven fast generation of robot behaviour tree based on large language model,” in 2024 2nd International Conference on Foundation and Large Language Models (FLLM). IEEE, 2024, pp. 392–397
2024
-
[29]
Fine-tuning vision-language model for automated engineering drawing information extraction,
M. T. Khan, L. Chen, Y . H. Ng, W. Feng, N. Y . J. Tan, and S. K. Moon, “Fine-tuning vision-language model for automated engineering drawing information extraction,” arXiv preprint arXiv:2411.03707 , 2024
2024 arXiv
-
[30]
Personal llm agents: Insights and sur- vey about the capability, efficiency and security,
Y . Li, H. Wen, W. Wang, X. Li, Y . Yuan, G. Liu, J. Liu, W. Xu, X. Wang, Y . Sun et al. , “Personal llm agents: Insights and sur- vey about the capability, efficiency and security,” arXiv preprint arXiv:2401.05459, 2024
2024 arXiv
-
[31]
Questioning, answering, and captioning for zero-shot detailed image caption,
D.-T. Luu, V .-T. Le, and D. M. V o, “Questioning, answering, and captioning for zero-shot detailed image caption,” in Proceedings of the Asian Conference on Computer Vision , 2024, pp. 242–259
2024
-
[32]
Right this way: Can vlms guide us to see more to answer questions?
L. Liu, D. Yang, S. Zhong, K. S. S. Tholeti, L. Ding, Y . Zhang, and L. Gilpin, “Right this way: Can vlms guide us to see more to answer questions?” Advances in Neural Information Processing Systems, vol. 37, pp. 132 946–132 976, 2024
2024
-
[33]
Cogvlm2: Visual language models for image and video understanding,
W. Hong, W. Wang, M. Ding, W. Yu, Q. Lv, Y . Wang, Y . Cheng, S. Huang, J. Ji, Z. Xue et al. , “Cogvlm2: Visual language models for image and video understanding,” arXiv preprint arXiv:2408.16500, 2024
2024 arXiv
-
[34]
Physically grounded vision-language models for robotic manipulation,
J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh, “Physically grounded vision-language models for robotic manipulation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 12 462–12 469
2024
-
[35]
Lmeye: An interactive perception network for large language models,
Y . Li, B. Hu, X. Chen, L. Ma, Y . Xu, and M. Zhang, “Lmeye: An interactive perception network for large language models,” IEEE Transactions on Multimedia , 2024
2024
-
[36]
Chat with the environment: Interactive multimodal perception using large language models,
X. Zhao, M. Li, C. Weber, M. B. Hafez, and S. Wermter, “Chat with the environment: Interactive multimodal perception using large language models,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 3590–3596
2023
-
[37]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763
2021
-
[38]
Simvlm: Simple visual language model pretraining with weak supervision,
Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y . Tsvetkov, and Y . Cao, “Simvlm: Simple visual language model pretraining with weak supervision,” arXiv preprint arXiv:2108.10904 , 2021
2021 arXiv
-
[39]
Flamingo: a visual language model for few-shot learning,
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” Advances in neural information processing systems , vol. 35, pp. 23 716–23 736, 2022
2022
-
[40]
Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,
J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742
2023
-
[41]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologi...
2019
-
[42]
Llm-bt: Performing robotic adaptive tasks based on large language models and behavior trees,
H. Zhou, Y . Lin, L. Yan, J. Zhu, and H. Min, “Llm-bt: Performing robotic adaptive tasks based on large language models and behavior trees,” in 2024 IEEE International Conference on Robotics and Au- tomation (ICRA). IEEE, 2024, pp. 16 655–16 661
2024
-
[43]
Llm text generation for service robot context,
L. Q. Silva, A. P. F. Mascarenhas, M. A. Sim ˜oes, I. J. Rodowanski, J. A. P. De Campos, J. R. De Souza, and J. G. Da Silva Filho, “Llm text generation for service robot context,” in 2024 Brazilian Symposium on Robotics (SBR), and 2024 Workshop on Robotics in Education (WRE) ....
2024
-
[44]
Exploring the limits of transfer learning with a unified text-to-text transformer,
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020
2020
-
[45]
Large language models as zero-shot human models for human-robot interaction,
B. Zhang and H. Soh, “Large language models as zero-shot human models for human-robot interaction,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 7961–7968
2023
-
[46]
Foundation models in robotics: Applications, challenges, and the future,
R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y . Zhu, S. Song, A. Kapoor, K. Hausman et al. , “Foundation models in robotics: Applications, challenges, and the future,” The International Journal of Robotics Research , vol. 44, no. 5, pp. 701–739, 2025
2025
-
[47]
Enabling action crossmodality for a pretrained large language model,
A. Caesar, O. ¨Ozdemir, C. Weber, and S. Wermter, “Enabling action crossmodality for a pretrained large language model,” Natural Lan- guage Processing Journal, vol. 7, p. 100072, 2024
2024
-
[48]
Evaluating large language models trained on code,
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
2021 arXiv
-
[49]
Robocodex: Multimodal code generation for robotic behavior synthesis,
Y . Mu, J. Chen, Q. Zhang, S. Chen, Q. Yu, C. Ge, R. Chen, Z. Liang, M. Hu, C. Tao et al. , “Robocodex: Multimodal code generation for robotic behavior synthesis,” arXiv preprint arXiv:2402.16117 , 2024
2024 arXiv
-
[50]
Llms for coding and robotics education,
P. Shu, H. Zhao, H. Jiang, Y . Li, S. Xu, Y . Pan, Z. Wu, Z. Liu, G. Lu, L. Guan et al., “Llms for coding and robotics education,”arXiv preprint arXiv:2402.06116, 2024
2024 arXiv
-
[51]
3p-llm: Probabilistic path planning using large lan- guage model for autonomous robot navigation,
E. Latif, “3p-llm: Probabilistic path planning using large lan- guage model for autonomous robot navigation,” arXiv preprint arXiv:2403.18778, 2024
2024 arXiv
-
[52]
Integrating large language model for natural language-based instruction toward robust human-robot collaboration,
F. Gao, L. Xia, J. Zhang, S. Liu, L. Wang, and R. X. Gao, “Integrating large language model for natural language-based instruction toward robust human-robot collaboration,” Procedia CIRP, vol. 130, pp. 313– 318, 2024
2024
-
[53]
Applying large language model to a control system for multi-robot task assignment,
W. Zhao, L. Li, H. Zhan, Y . Wang, and Y . Fu, “Applying large language model to a control system for multi-robot task assignment,” Drones, vol. 8, no. 12, p. 728, 2024
2024
-
[54]
Training language models to follow instructions with human feedback,
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022
2022
-
[55]
Instruct2act: Mapping multi-modality instructions to robotic actions with large language model,
S. Huang, Z. Jiang, H. Dong, Y . Qiao, P. Gao, and H. Li, “Instruct2act: Mapping multi-modality instructions to robotic actions with large language model,” arXiv preprint arXiv:2305.11176 , 2023
2023 arXiv
-
[56]
An overview of robot embodied intelligence based on multimodal models: Tasks, models, and system schemes,
Y . Cong and H. Mo, “An overview of robot embodied intelligence based on multimodal models: Tasks, models, and system schemes,” International Journal of Intelligent Systems , vol. 2025, no. 1, p. 5124400, 2025
2025
-
[57]
Llama: Open and efficient foundation language models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[58]
Manipllm: Embodied multimodal large language model for object-centric robotic manipulation,
X. Li, M. Zhang, Y . Geng, H. Geng, Y . Long, Y . Shen, R. Zhang, J. Liu, and H. Dong, “Manipllm: Embodied multimodal large language model for object-centric robotic manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp....
2024
-
[59]
Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation,
J. Liu, M. Liu, Z. Wang, P. An, X. Li, K. Zhou, S. Yang, R. Zhang, Y . Guo, and S. Zhang, “Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation,” Advances in Neural Information Processing Systems , vol. 37, pp. 40 085–40 110, 2024
2024
-
[60]
Deployment of large language models to control mobile robots at the edge,
P. Sikorski, L. Schrader, K. Yu, L. Billadeau, J. Meenakshi, N. Mutha- rasan, F. Esposito, H. AliAkbarpour, and M. Babaias, “Deployment of large language models to control mobile robots at the edge,” in 2025 3rd International Conference on Mechatronics, Control and Robotics (I...
2025
-
[61]
Language, camera, autonomy! prompt-engineered robot control for rapidly evolving deployment,
J. P. Macdonald, R. Mallick, A. B. Wollaber, J. D. Pe ˜na, N. McNeese, and H. C. Siu, “Language, camera, autonomy! prompt-engineered robot control for rapidly evolving deployment,” in Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction , 2024, pp...
2024
-
[62]
Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,
J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,” Advances in neural information processing systems , vol. 32, 2019
2019
-
[63]
Transformer-based vision- language alignment for robot navigation and question answering,
H. Luo, Z. Guo, Z. Wu, F. Teng, and T. Li, “Transformer-based vision- language alignment for robot navigation and question answering,” Information Fusion, vol. 108, p. 102351, 2024
2024
-
[64]
Clip-rt: Learning language-conditioned robotic policies from natural language supervision,
G.-C. Kang, J. Kim, K. Shim, J. K. Lee, and B.-T. Zhang, “Clip-rt: Learning language-conditioned robotic policies from natural language supervision,” arXiv preprint arXiv:2411.00508 , 2024
2024 arXiv
-
[65]
Robotic-clip: Fine-tuning clip on action data for robotic applications,
N. Nguyen, M. N. Vu, T. D. Ta, B. Huang, T. V o, N. Le, and A. Nguyen, “Robotic-clip: Fine-tuning clip on action data for robotic applications,” arXiv preprint arXiv:2409.17727 , 2024
2024 arXiv
-
[66]
Roboclip: One demonstration is enough to learn robot policies,
S. Sontakke, J. Zhang, S. Arnold, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti, “Roboclip: One demonstration is enough to learn robot policies,” Advances in Neural Information Processing Systems , vol. 36, pp. 55 681–55 693, 2023
2023
-
[67]
Clip feature-based random- ized control using images and text for multiple tasks and robots,
K. Shibata, H. Deguchi, and S. Taguchi, “Clip feature-based random- ized control using images and text for multiple tasks and robots,” Advanced Robotics, vol. 38, no. 15, pp. 1066–1078, 2024
2024
-
[68]
Clip-fields: Weakly supervised semantic fields for robotic memory,
N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam, “Clip-fields: Weakly supervised semantic fields for robotic memory,” arXiv preprint arXiv:2210.05663 , 2022
2022 arXiv
-
[69]
Vision-language models as success detectors,
Y . Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi, “Vision-language models as success detectors,” arXiv preprint arXiv:2303.07280 , 2023
2023 arXiv
-
[70]
Roboflamingo-plus: Fusion of depth and rgb perception with vision-language models for enhanced robotic manipulation,
S. Wang, “Roboflamingo-plus: Fusion of depth and rgb perception with vision-language models for enhanced robotic manipulation,” arXiv preprint arXiv:2503.19510, 2025
2025 arXiv
-
[71]
Expanding frozen vision- language models without retraining: Towards improved robot percep- tion,
R. Tavassoli, M. Amani, and R. Akhavian, “Expanding frozen vision- language models without retraining: Towards improved robot percep- tion,” arXiv preprint arXiv:2308.16493 , 2023
2023 arXiv
-
[72]
Robotic environmental state recognition with pre-trained vision- language models and black-box optimization,
K. Kawaharazuka, Y . Obinata, N. Kanazawa, K. Okada, and M. In- aba, “Robotic environmental state recognition with pre-trained vision- language models and black-box optimization,” Advanced Robotics , vol. 38, no. 18, pp. 1255–1264, 2024
2024
-
[73]
Gpt-4 technical report,
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
2023 arXiv
-
[74]
The dawn of lmms: Preliminary explorations with gpt-4v (ision),
Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, “The dawn of lmms: Preliminary explorations with gpt-4v (ision),” arXiv preprint arXiv:2309.17421, vol. 9, no. 1, p. 1, 2023
2023 arXiv
-
[75]
Next-generation human- robot interaction with chatgpt and robot operating system,
A. Koubaa, A. Ammar, and W. Boulila, “Next-generation human- robot interaction with chatgpt and robot operating system,” Software: Practice and Experience , vol. 55, no. 2, pp. 355–382, 2025
2025
-
[76]
Semantic intelligence: Integrating gpt-4 with a planning in low-cost robotics,
J. Barkley, A. George, and A. B. Farimani, “Semantic intelligence: Integrating gpt-4 with a planning in low-cost robotics,” arXiv preprint arXiv:2505.01931, 2025
2025 arXiv
-
[77]
Dobby: a conversational service robot driven by gpt-4,
C. Stark, B. Chun, C. Charleston, V . Ravi, L. Pabon, S. Sunkari, T. Mohan, P. Stone, and J. Hart, “Dobby: a conversational service robot driven by gpt-4,” in2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN) . IEEE, 2024, pp. 1362–1369
2024
-
[78]
Exploring gpt-4 for robotic agent strategy with real-time state feedback and a reactive behaviour framework,
T. O’Brien and Y . Sims, “Exploring gpt-4 for robotic agent strategy with real-time state feedback and a reactive behaviour framework,” arXiv preprint arXiv:2503.23601 , 2025
2025 arXiv
-
[79]
From text to motion: Grounding gpt-4 in a humanoid robot
T. Yoshida, A. Masumori, and T. Ikegami, “From text to motion: Grounding gpt-4 in a humanoid robot ”alter3”,” 2023. [Online]. Available: https://arxiv.org/abs/2312.06571
2023 arXiv
-
[80]
Improved baselines with visual instruction tuning,
H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” 2023
2023
-
[81]
Reasoning grasping via multimodal large language model,
S. Jin, J. Xu, Y . Lei, and L. Zhang, “Reasoning grasping via multimodal large language model,” arXiv preprint arXiv:2402.06798 , 2024
2024 arXiv
-
[82]
Robo2vlm: Visual question answering from large-scale in-the-wild robot manipulation datasets,
K. Chen, S. Xie, Z. Ma, and K. Goldberg, “Robo2vlm: Visual question answering from large-scale in-the-wild robot manipulation datasets,” arXiv preprint arXiv:2505.15517 , 2025
2025 arXiv
-
[83]
Palm-e: An embodied multimodal language model,
D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang et al., “Palm-e: An embodied multimodal language model,” 2023
2023
-
[84]
Palm: Scaling language modeling with pathways,
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmannet al., “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023
2023
-
[85]
Evalu- ating llms for code generation in hri: A comparative study of chatgpt, gemini, and claude,
A. Sobo, A. Mubarak, A. Baimagambetov, and N. Polatidis, “Evalu- ating llms for code generation in hri: A comparative study of chatgpt, gemini, and claude,” Applied Artificial Intelligence , vol. 39, no. 1, p. 2439610, 2025
2025
-
[86]
Embodied ai in mobile robots: Coverage path planning with large language models,
X. Kong, W. Zhang, J. Hong, and T. Braunl, “Embodied ai in mobile robots: Coverage path planning with large language models,” arXiv preprint arXiv:2407.02220, 2024
2024 arXiv
-
[87]
Task specific evaluation of large language models: A study for human-robot interaction,
A. Shenawa, “Task specific evaluation of large language models: A study for human-robot interaction,” 2025. KHAN et al.: FOUNDATION MODEL DRIVEN ROBOTICS: A COMPREHENSIVE REVIEW 15
2025
-
[88]
Gem- ini: a family of highly capable multimodal models,
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al. , “Gem- ini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023
2023 arXiv
-
[89]
Gr00t n1: An open foundation model for generalist humanoid robots,
J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang et al., “Gr00t n1: An open foundation model for generalist humanoid robots,” arXiv preprint arXiv:2503.14734 , 2025
2025 arXiv
-
[90]
Xie, Fundamentals of robotics: linking perception to action
M. Xie, Fundamentals of robotics: linking perception to action. World Scientific Publishing Company, 2003, vol. 54
2003
-
[91]
Active haptic perception in robots: a review,
L. Seminara, P. Gastaldo, S. J. Watt, K. F. Valyear, F. Zuher, and F. Mastrogiovanni, “Active haptic perception in robots: a review,” Frontiers in neurorobotics, vol. 13, p. 53, 2019
2019
-
[92]
A survey on coverage path planning for robotics,
E. Galceran and M. Carreras, “A survey on coverage path planning for robotics,” Robotics and Autonomous systems , vol. 61, no. 12, pp. 1258–1276, 2013
2013
-
[93]
Kurdila and P
A. Kurdila and P. Ben-Tzvi, Dynamics and Control of Robotic Systems. Wiley, 2019. [Online]. Available: https://books.google.com/ books?id=TOOqDwAAQBAJ
2019
-
[94]
Human–robot interaction: status and challenges,
T. B. Sheridan, “Human–robot interaction: status and challenges,” Human factors, vol. 58, no. 4, pp. 525–532, 2016
2016
-
[95]
Deep learning and machine learning–object detection and semantic segmentation: From theory to applications,
J. Ren, Z. Bi, Q. Niu, J. Liu, B. Peng, S. Zhang, X. Pan, J. Wang, K. Chen, C. H. Yin et al., “Deep learning and machine learning–object detection and semantic segmentation: From theory to applications,” arXiv preprint arXiv:2410.15584 , 2024
-
[96]
Language-augmented symbolic planner for open-world task planning,
G. Chen, L. Yang, R. Jia, Z. Hu, Y . Chen, W. Zhang, W. Wang, and J. Pan, “Language-augmented symbolic planner for open-world task planning,” arXiv preprint arXiv:2407.09792 , 2024
2024 arXiv
-
[97]
Practical pid controller tuning for motion control,
O. Ozen, E. Sariyildiz, H. Yu, K. Ogawa, K. Ohnishi, and A. Sabanovic, “Practical pid controller tuning for motion control,” in 2015 IEEE International Conference on Mechatronics (ICM) . IEEE, 2015, pp. 240–245
2015
-
[98]
From grounding to manipulation: Case studies of founda- tion model integration in embodied robotic systems,
X. Sui, D. Tian, Q. Sun, R. Chen, D. Choi, K. Kwok, and S. Po- ria, “From grounding to manipulation: Case studies of founda- tion model integration in embodied robotic systems,” arXiv preprint arXiv:2505.15685, 2025
2025
-
[99]
Open-world object manipulation using pre-trained vision-language models,
A. Stone, T. Xiao, Y . Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, S. Kirmani, B. Zitkovich, F. Xiaet al., “Open-world object manipulation using pre-trained vision-language models,” arXiv preprint arXiv:2303.00905, 2023
2023 arXiv
-
[100]
Foundation models defining a new era in vision: a survey and outlook,
M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: a survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2025
2025
-
[101]
Binding touch to everything: Learning unified multimodal tactile representations,
F. Yang, C. Feng, Z. Chen, H. Park, D. Wang, Y . Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens et al. , “Binding touch to everything: Learning unified multimodal tactile representations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...
2024
-
[102]
Octopi: Object property reasoning with large tactile-language models,
S. Yu, K. Lin, A. Xiao, J. Duan, and H. Soh, “Octopi: Object property reasoning with large tactile-language models,” arXiv preprint arXiv:2405.02794, 2024
2024 arXiv
-
[103]
A review of motion planning algorithms for intelligent robots,
C. Zhou, B. Huang, and P. Fr ¨anti, “A review of motion planning algorithms for intelligent robots,” Journal of Intelligent Manufacturing, vol. 33, no. 2, pp. 387–424, 2022
2022
-
[104]
Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,
D. Shah, B. Osi ´nski, S. Levine et al., “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,” inConference on robot learning . PMLR, 2023, pp. 492–504
2023
-
[105]
Scaling vision transformers to 22 billion parameters,
M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin et al., “Scaling vision transformers to 22 billion parameters,” in International Confer- ence on Machine Learning . PMLR, 2023, pp. 7480–7512
2023
-
[106]
A generalist agent,
S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y . Sulsky, J. Kay, J. T. Springenberg et al., “A generalist agent,” arXiv preprint arXiv:2205.06175 , 2022
2022 arXiv
-
[107]
Rt-1: Robotics transformer for real-world control at scale,
A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022
2022 arXiv
-
[109]
Genchip: Generating robot policy code for high-precision and contact- rich manipulation tasks,
K. Burns, A. Jain, K. Go, F. Xia, M. Stark, S. Schaal, and K. Hausman, “Genchip: Generating robot policy code for high-precision and contact- rich manipulation tasks,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2024, pp. 9596– 9603
2024
-
[110]
Genswarm: Scalable multi-robot code-policy generation and deployment via language models,
W. Ji, H. Chen, M. Chen, G. Zhu, L. Xu, R. Groß, R. Zhou, M. Cao, and S. Zhao, “Genswarm: Scalable multi-robot code-policy generation and deployment via language models,” arXiv preprint arXiv:2503.23875 , 2025
2025
-
[111]
Chatgpt for robotics: Design principles and model abilities,
S. H. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “Chatgpt for robotics: Design principles and model abilities,” Ieee Access, 2024
2024
-
[112]
Pivot: Iterative visual prompting elicits actionable knowledge for vlms,
S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu et al. , “Pivot: Iterative visual prompting elicits actionable knowledge for vlms,” arXiv preprint arXiv:2402.07872, 2024
2024 arXiv
-
[113]
Embodied large language models enable robots to complete complex tasks in unpredictable environments,
R. Mon-Williams, G. Li, R. Long, W. Du, and C. G. Lucas, “Embodied large language models enable robots to complete complex tasks in unpredictable environments,” Nature Machine Intelligence , pp. 1–10, 2025
2025
-
[114]
Hallucination is inevitable: An innate limitation of large language models,
Z. Xu, S. Jain, and M. Kankanhalli, “Hallucination is inevitable: An innate limitation of large language models,” arXiv preprint arXiv:2401.11817, 2024
2024 arXiv
-
[115]
Highlighting the safety concerns of deploying llms/vlms in robotics,
X. Wu, S. Chakraborty, R. Xian, J. Liang, T. Guan, F. Liu, B. M. Sadler, D. Manocha, and A. S. Bedi, “Highlighting the safety concerns of deploying llms/vlms in robotics,” arXiv preprint arXiv:2402.10340, 2024
2024 arXiv
-
[116]
A survey of robotic language grounding: Tradeoffs between symbols and embeddings,
V . Cohen, J. X. Liu, R. Mooney, S. Tellex, and D. Watkins, “A survey of robotic language grounding: Tradeoffs between symbols and embeddings,” arXiv preprint arXiv:2405.13245 , 2024
2024 arXiv
-
[117]
Affective feedback in closed loop human-robot interaction,
P. Rani, C. Liu, and N. Sarkar, “Affective feedback in closed loop human-robot interaction,” in Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-robot Interaction , 2006, pp. 335–336
2006
-
[118]
Humanoid robots in managerial positions-decision-making process and human oversight
I. Skubis and K. Wodarski, “Humanoid robots in managerial positions-decision-making process and human oversight.” Scientific Papers of Silesian University of Technology. Organization & Man- agement/Zeszyty Naukowe Politechniki Slaskiej. Seria Organizacji i Zarzadzanie, no. 189, 2023
2023
-
[119]
Towards audio language modeling–an overview,
H. Wu, X. Chen, Y .-C. Lin, K.-w. Chang, H.-L. Chung, A. H. Liu, and H.-y. Lee, “Towards audio language modeling–an overview,”arXiv preprint arXiv:2402.13236, 2024
2024 arXiv
-
[120]
Cross-modal safety mechanism transfer in large vision-language models,
S. Xu, L. Pang, Y . Zhu, H. Shen, and X. Cheng, “Cross-modal safety mechanism transfer in large vision-language models,” arXiv preprint arXiv:2410.12662, 2024
2024 arXiv
-
[121]
A survey of robot intelligence with large language models,
H. Jeong, H. Lee, C. Kim, and S. Shin, “A survey of robot intelligence with large language models,” Applied Sciences, vol. 14, no. 19, p. 8868, 2024
2024
-
[122]
Toward general-purpose robots via foundation models: A survey and meta-analysis,
Y . Hu, Q. Xie, V . Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y . Xie, T. Zhang, H.-S. Fang et al. , “Toward general-purpose robots via foundation models: A survey and meta-analysis,” arXiv preprint arXiv:2312.08782, 2023
2023 arXiv
-
[123]
Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding,
J. Jones, O. Mees, C. Sferrazza, K. Stachowicz, P. Abbeel, and S. Levine, “Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding,” arXiv preprint arXiv:2501.04693, 2025
2025 arXiv
-
[124]
Safety guardrails for llm-enabled robots,
Z. Ravichandran, A. Robey, V . Kumar, G. J. Pappas, and H. Has- sani, “Safety guardrails for llm-enabled robots,” arXiv preprint arXiv:2503.07885, 2025
2025
-
[125]
Gen2sim: Scaling up robot learning in simulation with generative models,
P. Katara, Z. Xian, and K. Fragkiadaki, “Gen2sim: Scaling up robot learning in simulation with generative models,” in 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 6672–6679
2024
-
[126]
Proc4gem: Foundation models for physical agency through procedural generation,
Y . Lin, J. Humplik, S. H. Huang, L. Hasenclever, F. Romano, S. Sal- iceti, D. Zheng, J. E. Chen, C. Barros, A. Collister et al., “Proc4gem: Foundation models for physical agency through procedural generation,” arXiv preprint arXiv:2503.08593 , 2025
2025 arXiv
-
[127]
Holodeck: Language guided generation of 3d embodied ai environments,
Y . Yang, F.-Y . Sun, L. Weihs, E. VanderBilt, A. Herrasti, W. Han, J. Wu, N. Haber, R. Krishna, L. Liuet al., “Holodeck: Language guided generation of 3d embodied ai environments,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp...
2024
-
[128]
Gzscenic: Au- tomatic scene generation for gazebo simulator,
A. Afzal, C. L. Goues, and C. S. Timperley, “Gzscenic: Au- tomatic scene generation for gazebo simulator,” arXiv preprint arXiv:2104.08625, 2021
2021 arXiv
-
[129]
World creator,
A. Karavaev, “World creator,” 2025, accessed: 2025-06-15. [Online]. Available: https://github.com/AlexKaravaev/world-creator
2025
-
[131]
Creative robot tool use with large language models,
M. Xu, P. Huang, W. Yu, S. Liu, X. Zhang, Y . Niu, T. Zhang, F. Xia, J. Tan, and D. Zhao, “Creative robot tool use with large language models,” arXiv preprint arXiv:2310.13065 , 2023
2023 arXiv
-
[132]
Gensim: Generating robotic simulation tasks via large language models,
L. Wang, Y . Ling, Z. Yuan, M. Shridhar, C. Bao, Y . Qin, B. Wang, H. Xu, and X. Wang, “Gensim: Generating robotic simulation tasks via large language models,” arXiv preprint arXiv:2310.01361 , 2023
2023 arXiv
-
[133]
Roboscript: Code generation for free- form manipulation tasks across real and simulation,
J. Chen, Y . Mu, Q. Yu, T. Wei, S. Wu, Z. Yuan, Z. Liang, C. Yang, K. Zhang, W. Shao et al. , “Roboscript: Code generation for free- form manipulation tasks across real and simulation,” arXiv preprint arXiv:2402.14623, 2024
2024 arXiv
-
[134]
Robotic control via embodied chain-of-thought reasoning,
M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,” arXiv preprint arXiv:2407.08693, 2024
2024 arXiv
-
[135]
Ai2-thor: An inter- active 3d environment for visual ai,
E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y . Zhu et al. , “Ai2-thor: An inter- active 3d environment for visual ai,” arXiv preprint arXiv:1712.05474, 2017
2017 arXiv
-
[136]
Typefly: Flying drones with large language model,
G. Chen, X. Yu, N. Ling, and L. Zhong, “Typefly: Flying drones with large language model,” arXiv preprint arXiv:2312.14950 , 2023
2023 arXiv
-
[137]
Smart-llm: Smart multi-agent robot task planning using large language models,
S. S. Kannan, V . L. Venkatesh, and B.-C. Min, “Smart-llm: Smart multi-agent robot task planning using large language models,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 12 140–12 147
2024
-
[138]
Taking flight with dialogue: Enabling natural language control for px4-based drone agent,
S. K. Lim, M. J. Y . Chong, J. H. Khor, and T. Y . Ling, “Taking flight with dialogue: Enabling natural language control for px4-based drone agent,” arXiv preprint arXiv:2506.07509 , 2025
2025 arXiv
-
[139]
Physical ai spatial intelligence warehouse,
NVIDIA, “Physical ai spatial intelligence warehouse,” Hugging Face Dataset Repository, 2025, accessed: 2025- 06-15. [Online]. Available: https://huggingface.co/datasets/nvidia/ PhysicalAI-Spatial-Intelligence-Warehouse
2025
-
[140]
Foundation models for autonomous robots in unstructured environments,
H. Naderi, A. Shojaei, and L. Huang, “Foundation models for autonomous robots in unstructured environments,” arXiv preprint arXiv:2407.14296, 2024
2024 arXiv
-
[141]
Large language models as generalizable policies for embodied tasks,
A. Szot, M. Schwarzer, H. Agrawal, B. Mazoure, R. Metcalf, W. Tal- bott, N. Mackraz, R. D. Hjelm, and A. T. Toshev, “Large language models as generalizable policies for embodied tasks,” in The Twelfth International Conference on Learning Representations , 2023
2023
-
[142]
Embodied chain of action reasoning with multi-modal foundation model for humanoid loco- manipulation,
Y . Hao, G. C. R. Bethala, N. Pudasaini, H. Huang, S. Yuan, C. Wen, B. Huang, A. Nguyen, and Y . Fang, “Embodied chain of action reasoning with multi-modal foundation model for humanoid loco- manipulation,” arXiv preprint arXiv:2504.09532 , 2025
2025
-
[143]
Quadrupedgpt: Towards a versatile quadruped agent in open-ended worlds,
Y . Mei, Y . Wang, S. Zheng, and Q. Jin, “Quadrupedgpt: Towards a versatile quadruped agent in open-ended worlds,” arXiv preprint arXiv:2406.16578, 2024
2024 arXiv
-
[144]
Autort: Embodied foundation models for large scale orchestration of robotic agents,
M. Ahn, D. Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan, K. Hausman, B. Ichter, A. Irpan, N. Joshi, R. Julian et al. , “Autort: Embodied foundation models for large scale orchestration of robotic agents,” arXiv preprint arXiv:2401.12963 , 2024
2024
-
[145]
Hu- manoid world models: Open world foundation models for humanoid robotics,
M. Q. Ali, A. Sridhar, S. Matiana, A. Wong, and M. Al-Sharman, “Hu- manoid world models: Open world foundation models for humanoid robotics,” arXiv preprint arXiv:2506.01182 , 2025
2025 arXiv
-
[146]
Sim-to-real transfer in robotics: Addressing the gap between simulation and real-world performance,
N. Chukwurah, A. S. Adebayo, and O. O. Ajayi, “Sim-to-real transfer in robotics: Addressing the gap between simulation and real-world performance,” International Journal of Robotics and Simulation, vol. 6, no. 1, pp. 89–102, 2024
2024
-
[147]
Natural language can help bridge the sim2real gap,
A. Yu, A. Foote, R. Mooney, and R. Mart ´ın-Mart´ın, “Natural language can help bridge the sim2real gap,” arXiv preprint arXiv:2405.10020 , 2024
2024 arXiv
-
[148]
A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models,
L. Da, J. Turnau, T. P. Kutralingam, A. Velasquez, P. Shakarian, and H. Wei, “A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models,” arXiv preprint arXiv:2502.13187, 2025
2025 arXiv
-
[149]
From abstraction to reality: Darpa’s vision for robust sim-to-real autonomy,
E. Noorani, Z. Serlin, B. Price, and A. Velasquez, “From abstraction to reality: Darpa’s vision for robust sim-to-real autonomy,” arXiv preprint arXiv:2503.11007, 2025
2025 arXiv
-
[150]
R3m: A universal visual representation for robot manipulation,
S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” arXiv preprint arXiv:2203.12601, 2022
2022 arXiv
-
[151]
Synthetic vision: Training vision-language models to un- derstand physics,
V . Balazadeh, M. Ataei, H. Cheong, A. H. Khasahmadi, and R. G. Krishnan, “Synthetic vision: Training vision-language models to un- derstand physics,” arXiv preprint arXiv:2412.08619 , 2024
2024
-
[152]
Fetchbot: Object fetching in cluttered shelves via zero- shot sim2real,
W. Liu, Y . Wan, J. Wang, Y . Kuang, X. Shi, H. Li, D. Zhao, Z. Zhang, and H. Wang, “Fetchbot: Object fetching in cluttered shelves via zero- shot sim2real,” arXiv preprint arXiv:2502.17894 , 2025
2025 arXiv
-
[153]
Generative artificial intelligence in robotic manipulation: A survey,
K. Zhang, P. Yun, J. Cen, J. Cai, D. Zhu, H. Yuan, C. Zhao, T. Feng, M. Y . Wang, Q. Chenet al., “Generative artificial intelligence in robotic manipulation: A survey,” arXiv preprint arXiv:2503.03464 , 2025
2025 arXiv
-
[154]
Understanding domain ran- domization for sim-to-real transfer,
X. Chen, J. Hu, C. Jin, L. Li, and L. Wang, “Understanding domain ran- domization for sim-to-real transfer,” arXiv preprint arXiv:2110.03239 , 2021
2021 arXiv
-
[155]
Domain random- ization for simulation-based policy optimization with transferability assessment,
F. Muratore, F. Treede, M. Gienger, and J. Peters, “Domain random- ization for simulation-based policy optimization with transferability assessment,” in Conference on Robot Learning . PMLR, 2018, pp. 700–713
2018
-
[156]
Exploring generative ai for sim2real in driving data synthesis,
H. Zhao, Y . Wang, T. Bashford-Rogers, V . Donzella, and K. Debattista, “Exploring generative ai for sim2real in driving data synthesis,” in2024 IEEE Intelligent Vehicles Symposium (IV) . IEEE, 2024, pp. 3071– 3077
2024
-
[157]
Adept: Adaptive diffusion environment for policy transfer sim-to-real,
Y . Yu, J. Xu, and L. Liu, “Adept: Adaptive diffusion environment for policy transfer sim-to-real,” arXiv preprint arXiv:2506.01759 , 2025
2025 arXiv
-
[158]
Bridging the simulation-to-real gap of depth images for deep reinforcement learning,
Y . Jang, J. Baek, S. Jeon, and S. Han, “Bridging the simulation-to-real gap of depth images for deep reinforcement learning,” Expert Systems with Applications, vol. 253, p. 124310, 2024
2024
-
[159]
Real-world robot learning with masked visual pre-training,
I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Dar- rell, “Real-world robot learning with masked visual pre-training,” in Conference on Robot Learning . PMLR, 2023, pp. 416–426
2023
-
[160]
Bridging the sim2real gap: Vision encoder pre-training for visuomotor policy transfer,
S. Biruduganti, Y . Yardi, and L. Ankile, “Bridging the sim2real gap: Vision encoder pre-training for visuomotor policy transfer,” arXiv preprint arXiv:2501.16389, 2025
2025 arXiv
-
[161]
Sugar: Pre-training 3d visual representations for robotics,
S. Chen, R. Garcia, I. Laptev, and C. Schmid, “Sugar: Pre-training 3d visual representations for robotics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 18 049–18 060
2024
-
[162]
Dreureka: Language model guided sim-to-real transfer,
Y . J. Ma, W. Liang, H.-J. Wang, S. Wang, Y . Zhu, L. Fan, O. Bastani, and D. Jayaraman, “Dreureka: Language model guided sim-to-real transfer,” arXiv preprint arXiv:2406.01967 , 2024
2024 arXiv
-
[163]
Rlingua: Improving reinforcement learning sample efficiency in robotic manipulations with large language models,
L. Chen, Y . Lei, S. Jin, Y . Zhang, and L. Zhang, “Rlingua: Improving reinforcement learning sample efficiency in robotic manipulations with large language models,” IEEE Robotics and Automation Letters , 2024
2024
-
[164]
Swarm-gpt: Combining large language models with safe motion planning for robot choreography design,
A. Jiao, T. P. Patel, S. Khurana, A.-M. Korol, L. Brunke, V . K. Adajania, U. Culha, S. Zhou, and A. P. Schoellig, “Swarm-gpt: Combining large language models with safe motion planning for robot choreography design,” arXiv preprint arXiv:2312.01059 , 2023
2023 arXiv
-
[165]
Rag llms are not safer: A safety analysis of retrieval-augmented generation for large language models,
B. An, S. Zhang, and M. Dredze, “Rag llms are not safer: A safety analysis of retrieval-augmented generation for large language models,” arXiv preprint arXiv:2504.18041 , 2025
2025 arXiv
-
[166]
Quantifying the sim2real gap: Model-based verification and validation in autonomous ground systems,
A. Waheed, M. Areti, L. Gallantree, and Z. Hasnain, “Quantifying the sim2real gap: Model-based verification and validation in autonomous ground systems,” IEEE Robotics and Automation Letters , 2025
2025
-
[167]
Llm-controller: Dynamic robot control adaptation using large language models,
R. Zahedifar, M. S. Baghshah, and A. Taheri, “Llm-controller: Dynamic robot control adaptation using large language models,” Robotics and Autonomous Systems, vol. 186, p. 104913, 2025
2025
-
[168]
Towards efficient discrete controller synthesis: Semantics- aware stepwise policy design via llm,
Y . Ishimizu, J. Li, T. Yamauchi, S. Chen, J. Cai, T. Hirano, and K. Tei, “Towards efficient discrete controller synthesis: Semantics- aware stepwise policy design via llm,” in 2024 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) . IEEE, 2024, pp. 1–4
2024
-
[169]
Progprompt: program genera- tion for situated robot task planning using large language models,
I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: program genera- tion for situated robot task planning using large language models,” Autonomous Robots, vol. 47, no. 8, pp. 999–1012, 2023
2023
-
[170]
Multibody models generated from natural language,
J. Gerstmayr, P. Manzl, and M. Pieber, “Multibody models generated from natural language,” Multibody System Dynamics , vol. 62, no. 2, pp. 249–271, 2024
2024
-
[171]
Leveraging large language models for comprehensive locomotion control in humanoid robots design,
S. Sun, C. Li, Z. Zhao, H. Huang, and W. Xu, “Leveraging large language models for comprehensive locomotion control in humanoid robots design,” Biomimetic Intelligence and Robotics , vol. 4, no. 4, p. 100187, 2024
2024
-
[172]
Replan: Robotic replanning with perception and language models,
M. Skreta, Z. Zhou, J. L. Yuan, K. Darvish, A. Aspuru-Guzik, and A. Garg, “Replan: Robotic replanning with perception and language models,” arXiv preprint arXiv:2401.04157 , 2024
2024 arXiv
-
[173]
Long- horizon locomotion and manipulation on a quadrupedal robot with large language models,
Y . Ouyang, J. Li, Y . Li, Z. Li, C. Yu, K. Sreenath, and Y . Wu, “Long- horizon locomotion and manipulation on a quadrupedal robot with large language models,” arXiv preprint arXiv:2404.05291 , 2024
2024 arXiv
-
[174]
Compositional foundation models for hierarchical planning,
A. Ajay, S. Han, Y . Du, S. Li, A. Gupta, T. Jaakkola, J. Tenenbaum, L. Kaelbling, A. Srivastava, and P. Agrawal, “Compositional foundation models for hierarchical planning,” Advances in Neural Information Processing Systems, vol. 36, pp. 22 304–22 325, 2023
2023
-
[175]
Llm- empowered embodied agent for memory-augmented task planning in household robotics,
M. Glocker, P. H ¨onig, M. Hirschmanner, and M. Vincze, “Llm- empowered embodied agent for memory-augmented task planning in household robotics,” arXiv preprint arXiv:2504.21716 , 2025
2025 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.