REVIEW 31 cited by
SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics. We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a 'semantic search' for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an 'iterative replanning' pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and natural language instruction for a mobile manipulator robot to execute. We provide real robot video demonstrations on our project page https://sayplan.github.io.
Forward citations
Cited by 31 Pith papers
-
ANCHOR: Agentic Noise Creation Framework for Human Simulation and Denoising Recommendation
A recommender-denoising method that trains a noise recognizer on LLM-simulated user misbehaviors instead of relying on heuristic rules.
-
VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
Using a simple action-token adapter, nine VLMs are compared as robot policy backbones, showing general VLM ability transfers poorly to control and the vision encoder is the key bottleneck.
-
Planning from Point Clouds over Continuous Actions for Multi-object Rearrangement
A hybrid A* search over SE(3) point cloud transforms, with learned suggesters proposing which object to move and where, solves multi-object rearrangement without discretizing actions.
-
Hypothesis-driven Model Expansion under Uncertainty for Open-World Robot Planning
HUME lets robots generate, plan over, and actively verify object-centric hypotheses from foundation models so incomplete symbolic models become usable for open-world household tasks.
-
HarnessWAM: Bridging Prediction and Deliberation in World Action Models
HarnessWAM wraps a world-action model with an external VLM task manager, task graph, capability projection, and recovery loop, reaching 59.6% full-task success on RoboMemArena and 23.7% SR on RoboCerebra Ideal.
-
GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planning
GraphThink uses a task graph for LLM planning prompts, GRPO rewards, and plan verification, plus a scene-graph event-driven replanner, achieving SOTA ALFRED results and stronger long-horizon generalization than API LLMs.
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks
LLMs generate subtask decompositions and ACL task spaces so sparse-reward RL solves long-horizon manipulation better than dense human rewards on five LIBERO tasks.
-
CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models
Topology-aware attention over hierarchical scene graphs lets a 3D-LLM ground, caption, and answer questions across multi-room homes, with large gains on a new HM3D benchmark.
-
Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search
SCOUT matches LLM planners on open-world interactive object search by scoring 3D scene-graph nodes with lightweight models distilled from LLM relational priors, at far lower compute cost.
-
ANNIE: Be Careful of Your Robots
The authors build a safety-centered benchmark and attack method that induces vision-language-action robot policies to violate ISO-based safety rules in a majority of tested episodes.
-
Adversarial Attacks on Robotic Vision Language Action Models
Text-based adversarial suffixes can make OpenVLA robot policies elicit chosen target actions with over 90% success on one-hot targets and persist across rollout steps.
-
Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulation
A decentralized diffusion policy for two robot arms that aligns a learned consensus embedding across agents and uses theory-of-mind prediction to keep that embedding informative.
-
MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems
A two-step soft-prompt backdoor attack called Robo-Troj (listed as MuTRAP on arXiv) makes LLM-based robot planners emit malicious plans when hidden trigger words are present, with near-perfect attack success.
-
I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences
RONAR is an LLM-based framework that narrates a mobile robot's experiences in natural language, and its user studies show that these narrations help people localize and explain robot failures faster than raw video interfaces.
-
ETA: A New Agentic Paradigm for Embodied Tasks
A general-purpose LLM planner using only observe, mark_point, and move_to solves 90% of 130 LIBERO manipulation tasks when allowed five attempts per task, with no robot-policy training.
-
N2M: Bridging Navigation and Manipulation by Learning Pose Preference from Rollout
N2M predicts preferable base poses for manipulation policies from ego-centric point clouds, learned from rollouts, lifting success from 3% to 54% in the PnPCounterToCab task.
-
Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent
A robot replanner that compares scene graphs to successful demonstrations before each subtask, triggering LLM-based replanning on mismatch, raises task success in AI2-THOR.
-
DyNaVLM: Zero-Shot Vision-Language Navigation System with Dynamic Viewpoints and Self-Refining Graph Memory
A zero-shot VLM navigation policy with dynamic waypoint selection and graph memory reports state-of-the-art results among VLM-based methods on ObjectNav and GOAT-Bench, plus real-world tests on a quadruped.
-
Interpretable Locomotion Prediction in Construction Using a Memory-Driven LLM Agent With Chain-of-Thought Reasoning
An LLM agent with short-term and long-term memory improved weighted F1 for construction locomotion prediction from 0.73 to 0.90 on a self-collected dataset of 226 multimodal samples.
-
Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation
A robot framework combining GPT-4V planning with a 3D feature-field skill policy improves long-horizon kitchen manipulation accuracy over LLM baselines, according to small real-robot trials.
-
TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applications
TimelyLLM segments LLM-generated robot plans into executable pieces and schedules those pieces by urgency, reducing response delays for time-critical robot tasks.
-
Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning
The thesis demonstrates that combining implicit 3D scene representations with LLM-based reasoning, using text as an interface, yields strong performance on robotic perception and spatial language tasks.
-
An LLM-powered Natural-to-Robotic Language Translation Framework with Correctness Guarantees
NRTrans uses a small Robot Skill Language with a compiler and iterative error feedback to improve the success rate of LLM-generated robot control programs.
-
Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systems
A position paper arguing that Bayesian inference could become a key design principle for embodied AI in open physical worlds, using Sutton's search-and-learning lens to explain its current absence.
-
Grounding Language Models with Semantic Digital Twins for Robotic Planning
The system grounds an LLM's action plans in hand-built semantic rules about a simulated home and reports success on all 14 selected ALFRED tasks.
-
HiBerNAC: Hierarchical Brain-emulated Robotic Neural Agent Collective for Disentangling Complex Manipulation
HiBerNAC, a multi-agent 'brain-inspired' planner layered on a reactive VLA, is claimed to cut long-horizon task time by 23% and reach 12-31% success where VLA baselines fail, but the supporting data are inconsistent.
-
Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization
Rhet2Pix combines staged LLM prompt decomposition with a discounted PPO fine-tuning scheme for Stable Diffusion, claiming strong rhetorical text-to-image generation, but the quantitative evidence is circular and undefined.
-
Semantic Intelligence: Integrating GPT-4 with A Planning in Low-Cost Robotics
A hybrid system where GPT-4 selects and adjusts A* paths lets a cheap quadruped follow semantic instructions like avoiding a toxic spill or collecting a resource first, with 90-100% success on the authors' tests.
-
LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics
An offline LLM agent-orchestration system with RAG memory reaches up to 84.3% lenient task-planning accuracy and 91.3% memory-response validity on three household scenarios.
-
From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent
Manus AI is a review-style preprint that summarizes public marketing and media claims about the autonomous agent Manus, without contributing new experiments, data, or original analysis.
Discussion (0). Continue with ORCID to comment.