REVIEW 6 cited by
RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We explore how intermediate policy representations can facilitate generalization by providing guidance on how to perform manipulation tasks. Existing representations such as language, goal images, and trajectory sketches have been shown to be helpful, but these representations either do not provide enough context or provide over-specified context that yields less robust policies. We propose conditioning policies on affordances, which capture the pose of the robot at key stages of the task. Affordances offer expressive yet lightweight abstractions, are easy for users to specify, and facilitate efficient learning by transferring knowledge from large internet datasets. Our method, RT-Affordance, is a hierarchical model that first proposes an affordance plan given the task language, and then conditions the policy on this affordance plan to perform manipulation. Our model can flexibly bridge heterogeneous sources of supervision including large web datasets and robot trajectories. We additionally train our model on cheap-to-collect in-domain affordance images, allowing us to learn new tasks without collecting any additional costly robot trajectories. We show on a diverse set of novel tasks how RT-Affordance exceeds the performance of existing methods by over 50%, and we empirically demonstrate that affordances are robust to novel settings. Videos available at https://snasiriany.me/rt-affordance
Forward citations
Cited by 6 Pith papers
-
O$^3$Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation
A one-shot training regime with DINOv2-enriched point clouds and joint cross-attention predicts 3D object-to-object affordance maps that guide optimization-based robotic manipulation.
-
AimBot: A Simple Auxiliary Visual Cue to Enhance Spatial Awareness of Visuomotor Policies
Overlaying end-effector-derived shooting lines and reticles on RGB images consistently raises success rates of visuomotor policies, especially on long-horizon manipulation tasks.
-
Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision
A new 28,497-sample dataset of 6DoF object manipulation trajectories is automatically extracted from egocentric video, and vision-language models are trained to generate these trajectories from action descriptions.
-
SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Models
SwitchVLA trains a vision-language-action policy to handle mid-execution instruction changes by conditioning on contact state and a three-way behavior mode, using only existing single-task demonstrations.
-
You Only Teach Once: Learn One-Shot Bimanual Robotic Manipulation from Video Demonstrations
From one human hand demonstration, YOTO generates hundreds of robot demonstrations and trains a bimanual diffusion policy that outperforms ACT, DP, DP3, and EquiBot on five real-world tasks.
-
Context-Dependent Affordance Computation in Vision-Language Models
Seven different agent personas made a vision-language model describe the same COCO image with under 10% lexical overlap, which the paper interprets as >90% context-dependent affordance computation.
Discussion (0). Continue with ORCID to comment.