REVIEW 6 cited by
Collaborating with language models for embodied reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reasoning in a complex and ambiguous environment is a key goal for Reinforcement Learning (RL) agents. While some sophisticated RL agents can successfully solve difficult tasks, they require a large amount of training data and often struggle to generalize to new unseen environments and new tasks. On the other hand, Large Scale Language Models (LSLMs) have exhibited strong reasoning ability and the ability to to adapt to new tasks through in-context learning. However, LSLMs do not inherently have the ability to interrogate or intervene on the environment. In this work, we investigate how to combine these complementary abilities in a single system consisting of three parts: a Planner, an Actor, and a Reporter. The Planner is a pre-trained language model that can issue commands to a simple embodied agent (the Actor), while the Reporter communicates with the Planner to inform its next command. We present a set of tasks that require reasoning, test this system's ability to generalize zero-shot and investigate failure cases, and demonstrate how components of this system can be trained with reinforcement-learning to improve performance.
Forward citations
Cited by 6 Pith papers
-
GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming
A streaming XML function-token interface with multi-channel scheduling lets robots execute concurrent speech and motion while the LLM is still generating, reportedly beating native function calling 15/15 vs 6/15 on co...
-
MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation
MALMM, a three-agent LLM framework with per-step environment feedback, achieves 81% average success on nine RLBench tasks versus 50% for a single-agent LLM baseline, in zero-shot settings.
-
Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation
A robot framework combining GPT-4V planning with a 3D feature-field skill policy improves long-horizon kitchen manipulation accuracy over LLM baselines, according to small real-robot trials.
-
Improving Vision-Language-Action Model with Online Reinforcement Learning
Alternating online RL on a frozen vision-language backbone with supervised fine-tuning on collected successes improves a VLA policy's task success and generalization.
-
Embodied CoT Distillation From LLM To Off-the-shelf Agents
DeDer distills LLM chain-of-thought reasoning into a two-tier small-language-model policy (rationale writer plus planner) and reports state-of-the-art ALFRED success rates for small-model embodied agents.
-
Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systems
A position paper arguing that Bayesian inference could become a key design principle for embodied AI in open physical worlds, using Sutton's search-and-learning lens to explain its current absence.
Discussion (0). Continue with ORCID to comment.