Pith. sign in

REVIEW 8 cited by

A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.08643 v2 pith:E7OLPPMG submitted 2025-02-12 cs.RO cs.AIcs.CV

Shivansh Patel , Xinchen Yin , Wenlong Huang , Shubham Garg , Hooshang Nayyeri , Li Fei-Fei , Svetlana Lazebnik , Yunzhu Li This is my paper
classification cs.ROcs.AIcs.CV
keywords rewarditerativeikerkeypointsmanipulationmulti-steptasktasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Task specification for robotic manipulation in open-world environments is challenging, requiring flexible and adaptive objectives that align with human intentions and can evolve through iterative feedback. We introduce Iterative Keypoint Reward (IKER), a visually grounded, Python-based reward function that serves as a dynamic task specification. Our framework leverages VLMs to generate and refine these reward functions for multi-step manipulation tasks. Given RGB-D observations and free-form language instructions, we sample keypoints in the scene and generate a reward function conditioned on these keypoints. IKER operates on the spatial relationships between keypoints, leveraging commonsense priors about the desired behaviors, and enabling precise SE(3) control. We reconstruct real-world scenes in simulation and use the generated rewards to train reinforcement learning (RL) policies, which are then deployed into the real world-forming a real-to-sim-to-real loop. Our approach demonstrates notable capabilities across diverse scenarios, including both prehensile and non-prehensile tasks, showcasing multi-step task execution, spontaneous error recovery, and on-the-fly strategy adjustments. The results highlight IKER's effectiveness in enabling robots to perform multi-step tasks in dynamic environments through iterative reward shaping.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks

    cs.RO 2026-07 conditional novelty 6.0 of 10

    LLMs generate subtask decompositions and ACL task spaces so sparse-reward RL solves long-horizon manipulation better than dense human rewards on five LIBERO tasks.

  2. Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Continuous multi-view image-space keypoint trajectories plus per-camera equivariant augmentation beat strong 3D and image baselines on MimicGen and real UR5 tasks.

  3. PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation

    cs.RO 2026-02 conditional novelty 6.0 of 10

    AgenticLab's closed-loop planning-language pipeline lets different vision-language models drive a real robot, and benchmark tests show action-verification quality, not planning, determines long-horizon success.

  4. VLM4D: Towards Spatiotemporal Awareness in Vision Language Models

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    VLM4D benchmarks spatiotemporal reasoning in VLMs and finds large gaps versus humans, with proposed methods showing partial improvement.

  5. "Harmless to You, Hurtful to Me!": Investigating the Detection of Toxic Languages Grounded in the Perspective of Youth

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    The authors construct the first Chinese youth-toxicity dataset, show that youth and adult perceptions of toxic language diverge, and report that adding contextual meta information improves detection accuracy.

  6. T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A zero-training framework that adaptively selects spatial representation extractors per object and per task stage improves real-world robot manipulation success and efficiency over fixed-representation baselines.

  7. AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making

    cs.RO 2025-06 conditional novelty 6.0 of 10

    AntiGrounding lifts candidate robot trajectories into the VLM's visual space via multi-view rendering and structured VQA, and reports 57.5% average success across eight manipulation tasks, beating three intermediate-r...

  8. SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training

    cs.RO 2025-07 conditional novelty 5.0 of 10

    Simulation-pretrained policies, with digital-twin demos for critic bootstrapping and action proposals, cut real-world RL training time while reaching near-perfect success on three manipulation tasks.

Pith tools