Pith. sign in

REVIEW 3 major objections 6 minor 65 references

PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PartInstruct introduces the first large-scale benchmark for robot manipulation under part-level instructions and shows that even the strongest baseline, Gemini-2.0 Flash plus DP3-S, reaches only 31.72% success.

desk verdict A genuinely useful part-level manipulation benchmark that needs a feasibility-ceiling baseline before its numbers can be interpreted. read the letter →

arxiv 2505.21652 v3 pith:CCT33R3O submitted 2025-05-27 cs.RO cs.AI

Yifan Yin , Zhengtao Han , Shivam Aarya , Jianxin Wang , Shuhang Xu , Jiawei Peng , Angtian Wang , Alan Yuille
show 1 more author
Tianmin Shu
This is my paper
classification cs.ROcs.AI
keywords part-levelinstructionfollowingfine-grainedrobotmanipulationbenchmarkvision-languagepolicybi-levelplanning3Dpartsegmentationdiffusiongeneralizationevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PartInstruct advances the claim that fine-grained robot manipulation — following instructions that name object parts, such as "pick up the bottle and show me the cap" — requires part-level visual grounding and planning, and that no existing benchmark supplies both the training data and the evaluation to build such skills. To close that gap, the paper introduces a simulator-based benchmark with 513 annotated object instances across 14 categories, 1,302 tasks grouped into 16 classes, more than 10,000 expert demonstrations, and a five-way test suite that isolates generalization to new states, new object instances, new part combinations, new task types, and new object categories. Evaluating state-of-the-art methods, the paper finds that all of them struggle: the best bi-level planner (Gemini-2.0 Flash high-level planning with DP3-S low-level execution) reaches only 31.72% average success, the best end-to-end policy 15.40%, and zero-shot generalist policies achieve no success at all. The paper's constructive finding is that explicit part perception is the strongest available lever — adding 3D part segmentation to a diffusion policy more than doubles its success on ground-truth plans — while decomposing tasks into single-part base skills proves far more tractable than end-to-end learning.

What carries the argument

The load-bearing object is the benchmark itself, assembled from three coupled components. First, PartGym, a PyBullet-based simulator with a Franka Emika Panda robot, renders RGB, depth, scene point clouds, and ground-truth 2D and 3D part segmentations from PartNet Mobility assets totaling 513 instances carrying 4,653 part labels, including spatial parts such as 'left' and 'front' alongside semantic parts such as 'handle' and 'lid'. Second, a procedural demonstration pipeline: each task goal is a conjunction of goal predicates (GRASPING, FACING, TOUCHING, AT_POSITION), grasp poses come from the Volumetric Grasping Network, motions come from the BiRRT planner, and template instructions are paraphrased by GPT-4o into 3–8 language variants per task. Third, an evaluation protocol of five binary-success test sets that isolate generalization over object initial states, object instances, part combinations, task categories, and object categories. The mechanism carrying the paper's positive result is bi-level decomposition: a VLM high-level planner emits one base-skill instruction at a time and a 3D diffusion policy with an added part point-cloud channel (DP3-S) executes it, a combination that beats every end-to-end baseline on every test set.

What would settle it

Run the same oracle planner that generated the training demonstrations on all evaluation episodes and record its success rate: if the oracle fails on more than about five percent of episodes, the low policy scores cannot be attributed purely to model limitations. A complementary check is to have human annotators judge whether each paraphrased instruction is semantically equivalent to the executed skill chain on a random sample of episodes, which would settle whether failures come from misaligned language labels.

Watch

Extended reading notes

Core claim

PartInstruct's claim is that it is the first large-scale benchmark for training and evaluating robot manipulation models under part-level instructions, and that this capability is currently missing from the toolbox of vision-language robots. Each episode pairs an expert trajectory with a natural-language task instruction and a chain of base-skill subgoals over five skill types — grasp, touch, rotate, move gripper, and release — where every subgoal references at most one object part and success is decided by binary goal predicates such as GRASPING, FACING, and AT_POSITION. Its central empirical finding is that current models cannot robustly ground part concepts or predict actions in 3D: the best bi-level baseline reaches 31.72% average success across the five test sets, end-to-end policies stay below 16%, and performance collapses on novel task categories and long skill chains. Its second finding is that explicit 3D part segmentation is the biggest available ingredient: DP3-S with ground-truth part masks reaches 32.00% on ground-truth high-level plans versus roughly 13% for the same policy without part input, and the drop from 32.00% to 20.97% when the masks come from a SAM2-based grounding pipeline shows that part grounding is itself a live bottleneck.

Load-bearing premise

The benchmark's conclusions about model capability rest on the assumption that the automatically generated demonstrations — grasp poses from the Volumetric Grasping Network, motion plans from BiRRT, and GPT-4o paraphrases of template instructions — are physically correct, semantically faithful to their instructions, and diverse enough for training; the paper reports no oracle execution rate or human validation for this data, so low policy scores could partly reflect flawed or mislabelled demonstrations rather than task difficulty.

Editorial extensions

If this is right

  • Fine-grained part-level manipulation is unsolved by current generalist policies: RT-1, Octo, and OpenVLA achieve no success in zero-shot evaluation on PartInstruct, and trained end-to-end policies stay below 16% average success.
  • Part annotations belong in training data: adding a 3D part mask to the low-level policy more than doubles success on ground-truth plans (32.00% for DP3-S versus roughly 13% without), so future fine-grained manipulation datasets should ship 2D and 3D part labels.
  • Bi-level planning is the more tractable architecture: a VLM planner that emits single-part skill subgoals plus a learned low-level executor outperforms end-to-end learning in every one of the five test sets.
  • Long-horizon part manipulation is the hardest regime: the novel-task-category test, which requires chains of several skills, caps every baseline at about 12% or lower, showing that planning and execution errors accumulate.
  • Part grounding is a measurable bottleneck: the Florence-2 and SAM2 tracking pipeline reaches only 27.58% grounding success with 0.20 mean IoU, so stronger part-level perception should translate directly into higher task success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The demonstration set is generated with no reported oracle success rate or human validation, so a share of the measured policy failures could stem from physically flawed or semantically mismatched trajectories rather than model limits; a manual audit of a random episode sample would separate the two explanations.
  • If the grounding bottleneck holds beyond simulation, open-vocabulary 3D part segmentation becomes the highest-leverage research target: a testable prediction is that replacing the Florence-2 and SAM2 pipeline with a stronger grounding model would push DP3-S from 20.97% toward the ground-truth-mask ceiling of 32.00%.
  • The paper itself notes that its oracle-generated demonstrations limit behavioral diversity and that sim-to-real transfer is untested; with real human teleoperation data, the ranking of bi-level versus end-to-end methods could shift, since bi-level's advantage partly rides on clean oracle skill boundaries.
  • Spatial parts such as 'left' and 'right' change their referent as an object rotates, and the impact study shows they are among the hardest to execute; a dedicated test that checks whether a policy maintains the original spatial reference through a rotation would isolate this failure mode cleanly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. PartInstruct is a simulated benchmark and dataset for part-level, language-conditioned robot manipulation. It provides 513 object instances in 14 categories, 4,653 part labels, 1,302 tasks in 16 task classes, and over 10,000 expert demonstrations generated in a PyBullet-based simulator (PartGym) with VGN-based grasp detection, BiRRT motion planning, and template-based instructions paraphrased by GPT-4o. Each demonstration includes a task instruction, a chain of base skill instructions, and ground-truth 2D/3D part annotations. The paper evaluates six end-to-end vision-language policies and several bi-level VLM-planner/low-level-policy baselines across five generalization splits, and reports that the best baseline (Gemini-2.0 Flash + DP3-S) reaches only 31.72% average success while end-to-end policies stay below 16%, with 3D part masks giving large improvements. The paper's claim is that current models struggle to ground part concepts and act in 3D for long-horizon part-level tasks.

Significance. If the benchmark's data is reliable, PartInstruct fills an important gap: no existing large-scale instruction-following benchmark combines part-level task instructions, training demonstrations, 2D/3D part masks, and diverse object instances. The scale and breadth of the evaluation are substantial, with multiple strong baselines, five generalization splits, standard errors, and a useful ablation on 2D vs 3D part masks. The paper also ships a project website and appears to release the data and simulator. However, the central quantitative conclusion is not yet calibrated: the paper does not report an oracle success rate of the demonstration-generation pipeline on the test episodes, and it does not validate the GPT-4o paraphrases for semantic fidelity to the formal goal predicates. Until those are provided, the low success rates could partly reflect infeasible episodes or noisy instructions rather than model limitations, which tempers the strength of the 'models struggle' conclusion. These are fixable with additional experiments, and the paper would then be a solid benchmark contribution.

major comments (3)
  1. [III-C3, IV-B1, Tables V-VII] The paper never reports an oracle success rate for the demonstration-generation pipeline, which is the load-bearing calibration for the headline claim that state-of-the-art methods struggle. The pipeline described in Section III-C3 (VGN grasp detection [2] + BiRRT motion planning [20]) is used both to generate the training demonstrations and as the 'oracle motion planner' in the bi-level experiments of Section IV-B1, but no full-task success rate of this pipeline on the five test sets is reported, either with ground-truth high-level skill chains or with an oracle skill-completion checker (which the paper implicitly has, since Section IV-C1 says the oracle planner can decide when a subgoal is achieved). Without this number, the reader cannot distinguish between tasks that are genuinely hard for current models and test episodes that are unsolvable by the same VGN/BiRRT pipeline that generated the demonstrations, in which case all reported success rates are depressed by an uncontrolled feasibility factor. The discrepancy between Table V (Gemini-2.0 Flash + DP3-S reaches 31.72%) and Table VII (Gemini-2.0 Flash + the oracle motion planner reaches 22.62%) shows that the oracle executor is not a ceiling, and the explanation offered in Section IV-C1 (fixed-step updates let the learned policy correct mistakes) makes the need for a true oracle-ceiling experiment even more important. I request an additional experiment: ground-truth skill chains from the task definitions paired with the oracle motion planner, reporting per-test-set full-task success and per-skill success. This is a standard sanity check for a benchmark and is within the paper's existing scope.
  2. [III-C3, Table II, Appendix A3] The GPT-4o paraphrases of the template-based task instructions are not validated for semantic equivalence with the formal goal predicates. Section III-C3 states that GPT-4o is prompted to paraphrase each template, yielding 3-8 variants, but no human validation, automatic equivalence check, or error analysis is reported. If any paraphrase changes the intended part, direction, or ordering of subgoals, the corresponding episode is mislabeled and the benchmark becomes a test of paraphrase fidelity rather than part-level manipulation. Since the central claim is that current models 'struggle to robustly ground part concepts,' the benchmark's language channel must be trustworthy. I recommend a small human evaluation (e.g., three annotators per instruction, judging consistency with the original template and goal predicates) and a table of agreement rates, plus a few concrete examples of paraphrases in the main text or appendix.
  3. [Tables V and VI] There is an unexplained inconsistency between Table V and Table VI: the full bi-level system (Gemini-2.0 Flash + DP3-S) achieves 31.72% average success in Table V, while DP3-S paired with ground-truth high-level skill chains achieves only 20.97% in Table VI. This is surprising because the ground-truth skill chains should be at least as good as the VLM-generated ones. The paper does not discuss this comparison. The discrepancy could indicate that the learned low-level policy fails on certain parts of the ground-truth chains and that the VLM accidentally selects easier or shorter chains, or it could reflect a difference in evaluation protocol (e.g., the fixed-step update in the full system lets the policy re-plan after a failure, whereas the ground-truth-chain evaluation may commit to the full chain). Either way, it undermines the interpretability of the bi-level results and should be analyzed explicitly, for example by reporting per-skill success and per-test-set chain length statistics.
minor comments (6)
  1. [Table I] The symbols for whether a benchmark contains demonstrations, 2D part masks, and 3D part masks are not visible or are misrendered (e.g., the rows for CALVIN, VIMAbench, and LoHoRavens show empty or truncated entries). Please check the table rendering and provide a cleaner version.
  2. [Appendix D2] The zero-shot evaluation in Appendix D2 claims that RT-1, Octo, and OpenVLA 'fail to achieve any success,' but no table or per-model numbers are reported. Please add the quantitative results or at least a note on the number of rollouts.
  3. [Section IV-A2 and Table V] The 'All' column is not defined; please state whether it is the pooled success rate over all episodes or the average of the five test-set rates, and report the number of episodes per test set.
  4. [Section IV-B1] The fixed skill-update interval n (130 for grasp, 30 for move, 68 for touch, 40 for release, 22 for rotate) is a hyperparameter with no sensitivity analysis; please add a brief discussion or an ablation of n.
  5. [Figure 4] The heatmap labels for part names along the horizontal axis are extremely small and likely illegible in the printed version; please increase font size or rotate labels.
  6. [Reference list, Section IV-B1] The reference list contains an inconsistency in the SAM 2 citations: [34] is the SAM 2 paper and [35] is the Grounded SAM paper, but the text in Section IV-B1 says 'Grounded-SAM-2 [35]', which is not a quote from either reference. Please correct the citation to the appropriate Grounded-SAM-2 paper or clarify the relation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark results are independent empirical evaluations, with a missing oracle-ceiling confound but no derivation-by-construction.

full rationale

PartInstruct is a benchmark/resource paper; its central claims are (i) that a new benchmark exists with expert demonstrations and part-level annotations, and (ii) that current policies attain low success on it. Neither claim is derived by fitting a quantity to the same quantity it predicts. The success rates in Tables V-VIII and Figure 8 are obtained by training or prompting independent external methods (Octo, Act3D, RVT2, 3D-DA, DP, DP3, GPT-4o, Gemini) on the released data and measuring rollouts against the stated goal predicates; no benchmark parameter is fitted from those test rollouts and then renamed a prediction. The skill-instruction decomposition and success predicates in Section III-A are self-defined, as is standard for benchmark construction, but they do not force the measured outcomes: a sufficiently skilled policy could in principle succeed. The absence of a reported oracle-ceiling success rate is a real validity/interpretability concern—low scores could conflate task difficulty with VGN/BiRRT demonstration-generation failures—but it is not a circular reduction, because the evaluated policies are not constructed to reproduce the benchmark's own success criteria by definition. The few references to prior work by the authors' group are not load-bearing, and the citations [42,25] supporting the decision not to use a VLM completion-checker are not self-citations of this paper's authors. No uniqueness theorem is imported, and no ansatz is smuggled in via citation. Overall, no step in the paper's derivation is equivalent to its input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's claims are empirical, so the ledger holds dataset-construction assumptions rather than mathematical axioms. The main unquantified choices are the oracle success tolerances and the VLM update intervals; neither is fitted to test outcomes.

free parameters (2)
  • VLM skill update interval n = grasp_obj:130, move_gripper:30, touch_obj:68, release_gripper:40, rotate_obj:22 (steps)
    Set in Appendix D3 as average skill lengths in the training set. It controls how often the high-level planner can revise its subgoal but is not fitted to test performance.
  • Goal predicate success tolerances = not reported
    The oracle uses binary criteria for FACING, AT_POSITION, TOUCHING and related predicates (Appendix A), but exact numeric tolerances are not given. These hand-set thresholds determine every reported success rate.
assumptions (4)
  • domain assumption PartNet Mobility part segmentations are correct and complete for all 14 object categories used in PartInstruct.
    Invoked in Section III-C1 when building the dataset on PartNet Mobility. If labels are wrong, part grounding training and evaluation are compromised.
  • domain assumption GPT-4o paraphrasing preserves the semantics of template task instructions.
    Section III-C3 uses GPT-4o to generate 3-8 variants per template. No human or oracle check is reported to confirm the paraphrases keep the original goal predicates.
  • domain assumption The oracle planner (VGN grasp detection plus BiRRT) generates feasible expert demonstrations, and PyBullet is a faithful model of rigid-body manipulation.
    Section III-C3. All training demonstrations and oracle evaluations depend on this, and Section VI acknowledges that sim-to-real transfer is not validated.
  • domain assumption Goal predicate success criteria are a valid operationalization of the natural-language tasks.
    Section III-A defines goals via predicates such as FACING, GRASPING, and AT_POSITION. The mapping from language to predicates is assumed correct and is not independently verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation." pith.science (2026). https://pith.science/paper/CCT33R3O

@misc{pith2026250521652,
  author       = {Pith},
  title        = {Pith review of: PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCT33R3O}},
  note         = {Machine review of arXiv:2505.21652}
}
read the original abstract

Fine-grained robot manipulation, such as lifting and rotating a bottle to display the label on the cap, requires robust reasoning about object parts and their relationships with intended tasks. Despite recent advances in training general-purpose robot manipulation policies guided by language instructions, there is a notable lack of large-scale datasets for fine-grained manipulation tasks with part-level instructions and diverse 3D object instances annotated with part-level labels. In this work, we introduce PartInstruct, the first large-scale benchmark for training and evaluating fine-grained robot manipulation models using part-level instructions. PartInstruct comprises 513 object instances across 14 categories, each annotated with part-level information, and 1302 fine-grained manipulation tasks organized into 16 task classes. Our training set consists of over 10,000 expert demonstrations synthesized in a 3D simulator, where each demonstration is paired with a high-level task instruction, a chain of base part-based skill instructions, and ground-truth 3D information about the object and its parts. Additionally, we designed a comprehensive test suite to evaluate the generalizability of learned policies across new states, objects, and tasks. We evaluated several state-of-the-art robot manipulation approaches, including end-to-end vision-language policy learning and bi-level planning models for robot manipulation on our benchmark. The experimental results reveal that current models struggle to robustly ground part concepts and predict actions in 3D space, and face challenges when manipulating object parts in long-horizon tasks.

Figures

Figures reproduced from arXiv: 2505.21652 by the authors.

Figure 1
Figure 1. An example fine-grained robot manipulation task in PartInstruct. To successfully perform the task described in the instruction (e.g., showing the cap without occluding it), the robot needs to reason about what object parts are relevant, ground the parts to its 3D visual perception, and plan for a sequence of part-level manipulation skills (e.g., the bottom sequence). Native object manipulation without a detailed und… view at source ↗
Figure 2
Figure 2. Example tasks and expert demonstrations in the dataset. Each task is defined by a task instruction. Each demonstration [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. PartGym supports multimodal observations, including RGB images, depth maps, and scene point clouds (PCDs). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Number of object instances in each object category. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Representative object assets from PartInstruct. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Overview of the bi-level planning framework. The High-Level Task Planner generates a skill instruction as a subgoal [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Success Rates of all baselines. The left group [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Selected Visual Modalities in PartGym. 2) Key Features of PartGym: The aim of PartGym is to boost embodied AI research related to interaction with table-top object parts. PartGym supports real-time rendering of different visual modalities (see [PITH_FULL_IMAGE:figures…
Figure 10
Figure 10. Figure 10: Left: Training set. Right: Test 1(OS) [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Left: Training set. Right: Test 2(OI) [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Above: Training set. Below: Test 3(TP) [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Above: Training set. Below: Test 4(TC) [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Left: Training set. Right: Test 5(OC) [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Parts in PartInstruct episodes, grouped by seen object types. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Parts in PartInstruct episodes, grouped by unseen object types. [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 28 canonical work pages

  1. [2]

    V olumetric grasping network: Real- time 6 dof grasp detection in clutter

    Michel Breyer, Jen Jen Chung, Lionel Ott, Roland Sieg- wart, and Juan Nieto. V olumetric grasping network: Real- time 6 dof grasp detection in clutter. InConference on Robot Learning, pages 1602–1611. PMLR, 2021

    work page 2021

  2. [20]

    Rrt-connect: An efficient approach to single-query path planning

    James J Kuffner and Steven M LaValle. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065), volume 2, pages 995–1001. IEEE, 2000

    work page 2000

  3. [1]

    Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Haus- man, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

  4. [3]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    arXiv 2022

  5. [4]

    Shapenet: An information-rich 3d model repository.arXiv preprint arXiv:1512.03012, 2015

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository.arXiv preprint arXiv:1512.03012, 2015

    arXiv 2015

  6. [5]

    Dif- fusion policy: Visuomotor policy learning via action diffusion.arXiv preprint arXiv:2303.04137, 2023

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- fusion policy: Visuomotor policy learning via action diffusion.arXiv preprint arXiv:2303.04137, 2023

    arXiv 2023

  7. [6]

    Pybullet, a python module for physics simulation for games, robotics and machine learning

    Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org, 2016–2021

    2016

  8. [7]

    Open6dor: Benchmarking open- instruction 6-dof object rearrangement and a vlm-based approach

    Yufei Ding, Haoran Geng, Chaoyi Xu, Xiaomeng Fang, Jiazhao Zhang, Songlin Wei, Qiyu Dai, Zhizheng Zhang, and He Wang. Open6dor: Benchmarking open- instruction 6-dof object rearrangement and a vlm-based approach. InFirst Vision and Language for Autonomous Driving and Robotics Workshop, 2024

    work page 2024

Show all 65 references
  1. [8]

    Self- supervised correspondence in visuomotor policy learn- ing.IEEE Robotics and Automation Letters, 5(2):492– 499, 2019

    Peter Florence, Lucas Manuelli, and Russ Tedrake. Self- supervised correspondence in visuomotor policy learn- ing.IEEE Robotics and Automation Letters, 5(2):492– 499, 2019

    2019

  2. [9]

    Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations

    Haoran Geng, Ziming Li, Yiran Geng, Jiayi Chen, Hao Dong, and He Wang. Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 2978–2988, 2023

    2023

  3. [10]

    Sage: Bridging semantic and actionable parts for generalizable articulated-object manipulation under language instructions.arXiv preprint arXiv:2312.01307, 2023

    Haoran Geng, Songlin Wei, Congyue Deng, Bokui Shen, He Wang, and Leonidas Guibas. Sage: Bridging semantic and actionable parts for generalizable articulated-object manipulation under language instructions.arXiv preprint arXiv:2312.01307, 2023

    2023 arXiv

  4. [11]

    Act3d: 3d feature field transform- ers for multi-task robotic manipulation

    Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d feature field transform- ers for multi-task robotic manipulation. In7th Annual Conference on Robot Learning, 2023

    2023

  5. [12]

    Rvt: Robotic view transformer for 3d object manipulation

    Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. InConference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  6. [13]

    Rvt-2: Learning precise manipulation from few demonstrations.arXiv preprint arXiv:2406.08545, 2024

    Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu- Wei Chao, and Dieter Fox. Rvt-2: Learning precise manipulation from few demonstrations.arXiv preprint arXiv:2406.08545, 2024

    2024 arXiv

  7. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  8. [15]

    Gpt-4o: The cutting-edge advancement in multimodal llm.Authorea Preprints, 2024

    Raisa Islam and Owana Marzia Moushi. Gpt-4o: The cutting-edge advancement in multimodal llm.Authorea Preprints, 2024

    2024

  9. [16]

    Rlbench: The robot learning bench- mark & learning environment.IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning bench- mark & learning environment.IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020

    2020

  10. [17]

    Vima: Robot manipulation with multimodal prompts.arXiv preprint arXiv:2306.02060, 2023

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: Robot manipulation with multimodal prompts.arXiv preprint arXiv:2306.02060, 2023

    2023 arXiv

  11. [18]

    3d diffuser actor: Policy diffusion with 3d scene representations.arXiv preprint arXiv:2402.10885, 2024

    Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations.arXiv preprint arXiv:2402.10885, 2024

    2024 arXiv

  12. [19]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  13. [21]

    Code as policies: Language model programs for em- bodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for em- bodied control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    2023

  14. [22]

    Composable part-based manipulation.arXiv preprint arXiv:2405.05876, 2024

    Weiyu Liu, Jiayuan Mao, Joy Hsu, Tucker Hermans, Animesh Garg, and Jiajun Wu. Composable part-based manipulation.arXiv preprint arXiv:2405.05876, 2024

    2024 arXiv

  15. [23]

    Learning to generalize across long-horizon tasks from human demonstrations

    Ajay Mandlekar, Danfei Xu, Roberto Mart ´ın-Mart´ın, Silvio Savarese, and Li Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085, 2020

    2003 arXiv

  16. [24]

    Calvin: A benchmark for language- conditioned policy learning for long-horizon robot ma- nipulation tasks.IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022

    Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot ma- nipulation tasks.IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022

    2022

  17. [25]

    Gamevlm: A decision-making framework for robotic task planning based on visual language models and zero-sum games.arXiv preprint arXiv:2405.13751, 2024

    Aoran Mei, Jianhua Wang, Guo-Niu Zhu, and Zhongxue Gan. Gamevlm: A decision-making framework for robotic task planning based on visual language models and zero-sum games.arXiv preprint arXiv:2405.13751, 2024

    2024 arXiv

  18. [26]

    Chang, Li Yi, Subarna Tripathi, Leonidas J

    Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. InThe IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2019

    2019

  19. [27]

    Maniskill: Generalizable manipulation skill bench- mark with large-scale demonstrations.arXiv preprint arXiv:2107.14483, 2021

    Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill bench- mark with large-scale demonstrations.arXiv preprint arXiv:2107.14483, 2021

    2021 arXiv

  20. [28]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Ab- hishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In2024 IEEE International Conference on Robotics and A...

    2024

  21. [29]

    Qi, Li Yi, Hao Su, and Leonidas J

    Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space, 2017. URL https://arxiv.org/abs/ 1706.02413

    2017 arXiv

  22. [30]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. InInternational conference on ma- chine learning,...

    2021

  23. [31]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  24. [32]

    Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration

    Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau B¨ol¨oni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In2018 IEEE international conference on robotics and automation (ICRA), pages 3758–3765. I...

    2018

  25. [34]

    Sam 2: Segment anything in images and videos, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Rong- hang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R ¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Doll ´ar, and Christoph Fei...

    2024

  26. [35]

    Grounded sam: Assembling open-world models for di- verse visual tasks, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for di- verse visual tasks, 20...

    2024

  27. [36]

    A comprehensive overview of deep learning techniques for 3d point cloud classification and semantic segmentation.Machine Vision and Applications, 35(4): 67, 2024

    Sushmita Sarker, Prithul Sarker, Gunner Stone, Ryan Gorman, Alireza Tavakkoli, George Bebis, and Javad Sattarvand. A comprehensive overview of deep learning techniques for 3d point cloud classification and semantic segmentation.Machine Vision and Applications, 35(4): 67, 2024

    2024

  28. [37]

    Perceiver-actor: A multi-task transformer for robotic ma- nipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic ma- nipulation. InConference on Robot Learning, pages 785–

  29. [38]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3009, 2023

    2023

  30. [39]

    A review of point cloud segmentation for understanding 3d indoor scenes.Visual Intelligence, 2(1):14, 2024

    Yuliang Sun, Xudong Zhang, and Yongwei Miao. A review of point cloud segmentation for understanding 3d indoor scenes.Visual Intelligence, 2(1):14, 2024

    2024

  31. [40]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Bur- nell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  32. [41]

    Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    2024 arXiv

  33. [42]

    Vlm see, robot do: Human demo video to robot action plan via vision language model

    Beichen Wang, Juexiao Zhang, Shuwen Dong, Irving Fang, and Chen Feng. Vlm see, robot do: Human demo video to robot action plan via vision language model. arXiv preprint arXiv:2410.08792, 2024

    2024

  34. [43]

    Learning adaptive planning representations with natural language guidance

    Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S Siegel, Jiahai Feng, Noa Korneev, Joshua B Tenenbaum, and Jacob Andreas. Learning adaptive planning representations with natural language guidance. arXiv preprint arXiv:2312.08566, 2023

    2023 arXiv

  35. [44]

    Embodied task planning with large language models.arXiv preprint arXiv:2307.01848, 2023

    Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models.arXiv preprint arXiv:2307.01848, 2023

    2023 arXiv

  36. [45]

    Chang, Leonidas J

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. InThe IEEE Conference on Computer Vision and Pa...

    2020

  37. [46]

    Sapien: A simulated part-based interactive environment

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1109...

    2020

  38. [47]

    Robopoint: A vision- language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Ar- salan Mousavian, and Dieter Fox. Robopoint: A vision- language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    2024 arXiv

  39. [48]

    A survey of imitation learning: Al- gorithms, recent developments, and challenges.IEEE Transactions on Cybernetics, 2024

    Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Al- gorithms, recent developments, and challenges.IEEE Transactions on Cybernetics, 2024

    2024

  40. [49]

    3d diffusion policy.arXiv preprint arXiv:2403.03954, 2024

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy.arXiv preprint arXiv:2403.03954, 2024

    2024 arXiv

  41. [50]

    Lohoravens: A long-horizon language-conditioned benchmark for robotic tabletop manipulation.arXiv preprint arXiv:2310.12020, 2023

    Shengqiang Zhang, Philipp Wicke, L ¨utfi Kerem S ¸enel, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Barbara Plank, and Hinrich Sch ¨utze. Lohoravens: A long-horizon language-conditioned benchmark for robotic tabletop manipulation.arXiv preprint arXiv:2310.12020, 2023

    arXiv

  42. [51]

    Deep imitation learning for complex manipulation tasks from virtual reality teleoperation

    Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In2018 IEEE international conference on robotics and automation (ICRA), pages 5628–5635. IEEE...

    2018

  43. [53]

    Table IX: Definition of Predicates Predicate Description ON(obj, part, contact) Whetherobjis on thecontact

    Predicate Definitions:This subsection gives the definition of the basic predicates utilized by the motion planner. Table IX: Definition of Predicates Predicate Description ON(obj, part, contact) Whetherobjis on thecontact. TOUCHING(obj, part) Whether the gripper is in contact ...

  44. [54]

    Table X: Definition of Base Skills Skill Description Preconditions Effects grasp_obj(obj, part) Robot graspsobjatpart

    Skill Definitions:This subsection shows the detailed definition of the five skills. Table X: Definition of Base Skills Skill Description Preconditions Effects grasp_obj(obj, part) Robot graspsobjatpart. ON(table, obj); ∼GRASPING(obj); ∼TOUCHING(obj) GRASPING(obj, part) move_gr...

  45. [55]

    Task Definitions:This subsection shows the detailed definition of different task types in PartInstruct. Table XI: Seen Task Instructions and Goal States Seen (10) Order Example Task Instruction Goal States 1 Grasp theobjectby thepart GRASPING(gripper, part),ON(obj, table) 2 To...

  46. [56]

    Table XIII: Observation and Action Space details

    Observation and Action Space:Table XIII shows the observation and action space available inPartGym. Table XIII: Observation and Action Space details. Observation Space Static View - RGB300×300×3 Static View - Depth300×300 Static View - PCD3×1024 Static View - Semantic300×300 S...

  47. [57]

    PartGym supports real-time rendering of different visual modalities (see Figure 9)

    Key Features of PartGym:The aim of PartGym is to boost embodied AI research related to interaction with table-top object parts. PartGym supports real-time rendering of different visual modalities (see Figure 9). In addition to the typical modalities like RGB, depth, and object...

  48. [58]

    Additionally, PartGym provides a framework to implement bi-level planning models for part-level manipulation tasks in simulation environments

    simulation engine using the ground-truth part assets given byPartNet Mobility[45] [26] [4]. Additionally, PartGym provides a framework to implement bi-level planning models for part-level manipulation tasks in simulation environments. It provides a template skill instruction g...

  49. [59]

    Figure 10: Left: Training set

    Visualization of Test Splits:We provide the visualization of all 5 test sets in this section. Figure 10: Left: Training set. Right: Test 1(OS). Figure 11: Left: Training set. Right: Test 2(OI). Figure 12: Above: Training set. Below: Test 3(TP). Figure 13: Above: Training set. ...

  50. [60]

    Statistics of PartInstruct Episodes:We provided detailed statistics about parts within each object type. T op Right Left Lid Front Back Bottom 0 50 100 150 200 250 300 350 400 Count Box Back Left Right T op HandleFront Bottom 0 100 200 300 400 500 600 Count Mug Lid T opRightNe...

  51. [61]

    For RVT2 [12] and Octo [41], we implemented both fine-tuning of the pretrained models and training from scratch on our dataset

    Training Details in End-to-End Policy Learning:We trained the baseline models, including Diffusion Policy (DP) [5], 3D Diffusion Policy (DP3) [49], and Act3D [11], from scratch. For RVT2 [12] and Octo [41], we implemented both fine-tuning of the pretrained models and training ...

  52. [62]

    For RT-1, we followed the implementation of Open X-Embodiment project and used the releasedrt_1_x_tf_trained_for_002272480_stepcheckpoint for inference

    Zero-Shot Evaluation of the Generalist Policy:We selected several popular generalist policies, including RT-1, Octo, and OpenVLA, and evaluated their zero-shot performance on our test sets. For RT-1, we followed the implementation of Open X-Embodiment project and used the rele...

  53. [63]

    Design Details of Bi-Level Planning:We outline the bi-level planning pipeline’s implementation here as a supplement to Section IV-B. Implementation of the High-Level Task Planner:The high-level task planner features a skill inference mechanism that leverages comprehensive cont...

  54. [64]

    The executed skill instructions show the sequence of actions taken so far

    The task instruction helps you understand the overall task goal. The executed skill instructions show the sequence of actions taken so far

  55. [65]

    The TCP poses and images together illustrate the state transitions of the previous action

    The gripper state shows whether the gripper is open or closed. The TCP poses and images together illustrate the state transitions of the previous action

  56. [66]

    Move the box in the air towards the right while keeping in touch with the right, then put it down

    The current object state, relative to the gripper, and the object motion (from TCP and images) can help determine if the last action was successful. Skill Descriptions: 1.grasp obj: -Description:This skill grasps an object by a specific part. -Parameters: part grasp:The exact ...

  57. [2024]

    URL https://arxiv.org/abs/2408.00714

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.