REVIEW 33 cited by
RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
From rearranging objects on a table to putting groceries into shelves, robots must plan precise action points to perform tasks accurately and reliably. In spite of the recent adoption of vision language models (VLMs) to control robot behavior, VLMs struggle to precisely articulate robot actions using language. We introduce an automatic synthetic data generation pipeline that instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Compared to alternative approaches, our method requires no real-world data collection or human demonstration, making it much more scalable to diverse environments and viewpoints. In addition, RoboPoint is a general model that enables several downstream applications such as robot navigation, manipulation, and augmented reality (AR) assistance. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks. Project website: https://robo-point.github.io.
Forward citations
Cited by 33 Pith papers
-
VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
Using a simple action-token adapter, nine VLMs are compared as robot policy backbones, showing general VLM ability transfers poorly to control and the vision encoder is the key bottleneck.
-
GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert
A frozen, large-scale pretrained diffusion policy converts sparse 3D waypoints from a VLM into dense robot actions, enabling zero-shot reuse of the action expert on new tasks and environments.
-
Weakly-Supervised Learning of Dense Functional Correspondences
A weakly-supervised pipeline that distills VLM functional part knowledge and multi-view spatial structure into a model for dense cross-category functional correspondence, outperforming baselines on new synthetic and r...
-
Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation
BadSem shows that semantic mismatches between images and text can serve as stealthy backdoor triggers for VLMs, achieving near-perfect attack success with low poisoning rates.
-
PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation
PartInstruct is a new large-scale simulated benchmark with part-level language instructions and training demonstrations; current robot policies achieve at most 31.72% average success on it.
-
BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation
Adding stage-wise temporal and spatial memory to a heatmap-prediction 3D VLA policy yields strong results on memory-dependent manipulation benchmarks while keeping data efficiency.
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation
Dense per-frame intermediate representations (traces, masks, grasp poses, subtasks) improve embodied VQA, VLA action generation, and world-model video prediction in the new 230k-episode RoboInter-Data suite.
-
RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination
A unified transformer model generates language-and-image planning sequences for embodied tasks, and shows real-robot manipulation without large-scale action pretraining.
-
Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification
Projecting 3D gripper keypoints onto camera pixels and classifying those pixels yields millimeter-precise, multi-modal closed-loop manipulation faster than diffusion policies.
-
EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos
A full-stack system curates 9.6K hours of egocentric video into language-aligned action priors that, after robot post-training and DAgger, enable free-form steerable dexterous manipulation at ~75% success across 40+ tasks.
-
RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thought
Training a 4B vision-language model to emit identity-tracked, visually grounded reasoning anchors improves embodied spatial, multi-view, and pointing task performance over 7B baselines.
-
Structured Observation Language for Efficient and Generalizable Vision-Language Navigation
SOL-Nav encodes RGB-D observations as grid-organized text and uses a 0.6B text-embedding model with four classification heads to predict navigation action blocks, reporting SOTA/comparable R2R-CE/RxR-CE results with 1...
-
Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training
A 6K-question embodied-reasoning benchmark plus a flow-matching action tokenizer let one 3B vision-language model reason and manipulate better than continuous- or discrete-action VLA baselines.
-
Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics
A 3D-aware VLM, RoboTracer, generates metric-grounded spatial traces for robot manipulation using scale supervision and metric-sensitive reinforcement rewards.
-
SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL
A two-phase interactive RL framework (DIRL) lets a 3B VLM learn to coordinate multiple vision and robot tools, reaching top benchmark scores and 86% real-robot pick-and-place success.
-
SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
Dense scene-graph-grounded rewards let a 7B multimodal LLM trained on 7K synthetic questions beat SFT and sparse-RL baselines and outscore GPT-4o on average across 12 spatial/real-world benchmarks.
-
FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
A VLM trained on auto-generated failure trajectories with executable correction actions helps VLA models detect and fix manipulation errors, lifting success rates by up to 22.6 percentage points.
-
O$^3$Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation
A one-shot training regime with DINOv2-enriched point clouds and joint cross-attention predicts 3D object-to-object affordance maps that guide optimization-based robotic manipulation.
-
Robix: A Unified Model for Robot Interaction, Reasoning and Planning
A three-stage-trained VLM unifies robot planning and dialogue, and beats commercial VLMs on the authors' interactive-task benchmarks.
-
AimBot: A Simple Auxiliary Visual Cue to Enhance Spatial Awareness of Visuomotor Policies
Overlaying end-effector-derived shooting lines and reticles on RGB images consistently raises success rates of visuomotor policies, especially on long-horizon manipulation tasks.
-
PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation
PASG automatically extracts object keypoints and axes and couples them through a fine-tuned vision-language model to task semantics, claiming manipulation performance comparable to manual annotations.
-
Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
A dexterous VLA pretrained on a 2.5M-instance human hand motion dataset transfers skills to a real robot hand, outperforming baselines in manipulation tasks.
-
RoboBrain 2.0 Technical Report
RoboBrain 2.0, a 7B/32B embodied vision-language model built on Qwen2.5-VL, reports state-of-the-art or near-top scores on several spatial and temporal reasoning benchmarks for robotics.
-
Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation
Gondola generates multi-view segmentation-mask-grounded next-step plans for robotic manipulation and reports improved generalization on the GemBench benchmark over a prior LLM-based planner.
-
GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation
GenManip is a benchmark and simulation platform with LLM-generated scene graphs for testing how robot policies generalize to new instructions, layouts, and objects.
-
UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
UAD distills affordance knowledge from vision-language models and DINOv2 features into a lightweight task-conditioned model that predicts pixel-level manipulation regions and improves few-shot imitation learning gener...
-
VideoMolmo: Spatio-Temporal Grounding Meets Pointing
A video language model that conditions each frame on earlier frames via a temporal attention module, predicts text-requested object points, and uses SAM2-based bidirectional mask fusion to outperform prior models on v...
-
Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models
Embodied-R1.5 is an 8B EFM achieving SOTA on 16 of 24 embodied VLM benchmarks, fine-tunable to outperform leading VLAs, with claimed zero-shot real-robot generalization.
-
Hierarchical Vision-Language Planning for Multi-Step Humanoid Manipulation
A three-layer hierarchical system using a VLM planner and VLM skill monitor with imitation-learned skills and an RL tracking policy achieved 73% success on a real humanoid pick-and-place task.
-
OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
A vision-language model fine-tuned on 572K synthetic simulation examples improves open-world mobile manipulation action decisions and object grounding over GPT-4o, with 21.9% full-task success in simulation and 90% ac...
-
On the Dual-Use Dilemma in Physical Reasoning and Force
Adding Asimov-style safety prompts to vision-language models lowers both harmful and helpful force generation for contact-rich robotic tasks.
- CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
Discussion (0). Sign in to comment.