REVIEW 12 cited by
GenSim: Generating Robotic Simulation Tasks via Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data to demonstrate significant task-level generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GenSim, has two modes: goal-directed generation, wherein a target task is given to the LLM and the LLM proposes a task curriculum to solve the target task, and exploratory generation, wherein the LLM bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, on which we conduct supervised finetuning and evaluate several LLMs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world and outperform baselines by 25%. See the project website (https://liruiw.github.io/gensim) for code, demos, and videos.
Forward citations
Cited by 12 Pith papers
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation
Dense per-frame intermediate representations (traces, masks, grasp poses, subtasks) improve embodied VQA, VLA action generation, and world-model video prediction in the new 230k-episode RoboInter-Data suite.
-
PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution
A file-based operating-system layer with a session verifier and persistent memory improves embodied-agent task completion on game, simulated, and real-robot platforms without retraining policies.
-
LLM Trainer: Automated Robotic Data Generation via Demonstration Augmentation using LLMs
An LLM-based pipeline automatically augments one human demonstration into a large imitation-learning dataset, using Thompson sampling to pick the best annotation and beating expert-annotated baselines on most tasks.
-
GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming
A streaming XML function-token interface with multi-channel scheduling lets robots execute concurrent speech and motion while the LLM is still generating, reportedly beating native function calling 15/15 vs 6/15 on co...
-
RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
RobotSmith autonomously designs, 3D-prints, and uses task-specific tools for robotic manipulation, raising task success from 2.8% (no tool) to 50% in simulation.
-
GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation
GenManip is a benchmark and simulation platform with LLM-generated scene graphs for testing how robot policies generalize to new instructions, layouts, and objects.
-
ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
ManiTaskGen automatically generates diverse, feasible mobile manipulation tasks from any input scene, and uses them to benchmark and improve vision-language robot agents.
-
EvoCurr: Self-evolving Curriculum with Behavior Code Generation for Complex Decision-making
EvoCurr couples an LLM curriculum designer with an LLM code-generating solver, but its only reported success is 1 of 5 runs and no direct baseline is shown.
-
SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training
Simulation-pretrained policies, with digital-twin demos for critic bootstrapping and action proposals, cut real-world RL training time while reaching near-perfect success on three manipulation tasks.
-
LLM-based Property-based Test Generation for Guardrailing Cyber-Physical Systems
An LLM-based tool, ChekProp, extracts safety properties from CPS code and documentation and generates property-based tests that can serve as design-time checks and proposed runtime guardrails.
-
Foundation Model Driven Robotics: A Comprehensive Review
A review of foundation-model-driven robotics that synthesizes recent work across perception, planning, control, HRI, simulation, and sim-to-real transfer, and highlights open challenges.
Discussion (0). Continue with ORCID to comment.