Pith. sign in

REVIEW 10 cited by

Foundation Models in Robotics: Applications, Challenges, and the Future

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.07843 v1 pith:XW44HDW5 submitted 2023-12-13 cs.RO

Roya Firoozi , Johnathan Tucker , Stephen Tian , Anirudha Majumdar , Jiankai Sun , Weiyu Liu , Yuke Zhu , Shuran Song
show 7 more authors
Ashish Kapoor Karol Hausman Brian Ichter Danny Driess Jiajun Wu Cewu Lu Mac Schwager
This is my paper · ORCID
classification cs.RO
keywords modelsfoundationroboticsapplicationschallengesdatarobotautonomy
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper (Preliminary release. We are committed to further enhancing and updating this work to ensure its quality and relevance) can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning

    cs.LG 2025-06 conditional novelty 7.0 of 10

    A lifelong robot learning method that mixes a growing library of LoRA-style experts with a context router and replays router coefficients to achieve forward transfer with near-zero forgetting.

  2. FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    FELT predicts finger pressure maps from RGB images and uses them or their learned features to improve manipulation policies without real tactile sensors at deployment.

  3. Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    VLAJS augments PPO with sparse annealed VLA guidance through directional regularization to cut required interactions by over 50% on manipulation tasks and enable zero-shot sim-to-real transfer.

  4. INSIGHT: INference-time Sequence Introspection for Generating Help Triggers in Vision-Language-Action Models

    cs.RO 2025-10 conditional novelty 6.0 of 10

    Token-level uncertainty sequences from a VLA policy, classified by a small transformer, predict when a robot should request human help better than static uncertainty scores.

  5. UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation

    cs.RO 2025-06 conditional novelty 6.0 of 10

    UAD distills affordance knowledge from vision-language models and DINOv2 features into a lightweight task-conditioned model that predicts pixel-level manipulation regions and improves few-shot imitation learning gener...

  6. RoboBERT: An End-to-end Multimodal Robotic Manipulation Model

    cs.RO 2025-02 conditional novelty 6.0 of 10

    A two-stage trained vision-language-action diffusion policy with carefully selected data augmentations reaches mean episode lengths of 4.52 (ABCD to D) and 3.79 (ABC to D) on CALVIN.

  7. Versatile and Generalizable Manipulation via Goal-Conditioned Reinforcement Learning with Grounded Object Detection

    cs.RO 2025-07 reject novelty 4.0 of 10

    In simulation, ground-truth mask goal conditioning reaches about 90% grasp success in and out of distribution, but the proposed GroundingDINO-based pipeline fails to reproduce this when trained with detector-generated masks.

  8. Robot Context Protocol (RCP): A Runtime-Agnostic Interface for Agent-Aware Robot Control

    cs.RO 2025-06 reject novelty 3.0 of 10

    RCP is a proposed middleware-agnostic protocol exposing robot capabilities via read, write, execute, and subscribe operations over HTTP, WebSocket, and SSE, targeting agent-robot integration.

  9. Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research

    cs.RO 2025-06 accept novelty 1.0 of 10

    A perspective article reviews the state of using foundation models for laboratory automation and proposes a roadmap for fully autonomous experiments.

  10. CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity

    cs.RO 2025-06

Pith tools