Pith. sign in

REVIEW 4 major objections 5 minor 63 references

Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A mask-grounded multi-view planner beats the prior LLM-based planner on all four GemBench generalization levels.

desk verdict Gondola is a solid step forward for grounded VLM planning in manipulation, but the headline SOTA claim is undercut by an uncontrolled replanning difference with the baseline. read the letter →

arxiv 2506.11261 v1 pith:TS7NID7T submitted 2025-06-12 cs.RO cs.AIcs.CV

Shizhe Chen , Ricardo Garcia , Paul Pacaud , Cordelia Schmid This is my paper
classification cs.ROcs.AIcs.CV
keywords groundedvision-languageplanningroboticmanipulationmulti-viewperceptionsegmentationmasksgeneralizationlong-horizontasksRLBenchGemBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gondola is a vision-language planner that tries to make high-level manipulation planning visually grounded rather than language-only. Given a natural-language instruction and images from several cameras, it outputs the next action as text interleaved with segmentation masks that mark the object and the target location in every view, together with the text of previously executed plans. The authors train the model in the RLBench simulator on three self-constructed datasets—grounded planning traces, multi-view referring expressions, and pseudo long-horizon concatenations—and evaluate it on GemBench, which tests new locations, novel rigid objects, novel articulated objects, and unseen long-horizon tasks. They report that Gondola beats the prior LLM-based planner 3D-LOTUS++ at all four generalization levels, with the largest gains on novel rigid and articulated objects. The intended payoff is a generalizable planner that can follow instructions in unseen scenes without retraining per object or task.

What carries the argument

The central object is the mask-grounded plan $(a, o, M_o, l, M_l)$—an action name paired with dense per-view segmentation masks for the manipulated object and for the target location. This representation carries the argument because it turns planning into a grounding task: the same model predicts both the semantic next step and the pixel-level referents that a downstream motion planner needs, instead of relying on coarse boxes, points, or captions. The enabling mechanisms are the <seg> token in the LLM vocabulary, which passes a hidden embedding to SAM2's mask decoder, and the concatenation of multi-view image tokens plus a compact text history of prior plans. Training on three constructed RLBench datasets—grounded planning, multi-view referring expressions, and pseudo long-horizon concatenations—is what makes the model learn to use the mechanism at inference time.

What would settle it

Run both systems with identical planning frequency—for example, both replanning at every step—and compare success rates on GemBench Levels 2 and 3; if the reported 10-point gaps disappear, the closed-loop replanning loop, not the mask-grounded planner, is carrying the improvement.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that converting high-level planning into a dense multi-view grounding problem improves generalization more than planning from language alone or from single-view grounding. Gondola produces a structured plan $(a, o, M_o, l, M_l)$: an action name, an object description with binary masks across all camera views, and a target location description with masks. The model is an InternVL-4B LLM fine-tuned with LoRA on the Sa2VA dense-grounding framework, and a dedicated <seg> token makes the LLM's hidden state prompt SAM2 to segment each referred object in each view. In the GemBench test split, Gondola achieves success rates of 87.3%, 74.8%, 52.4%, and 19.0% at Levels 1–4, against 68.7%, 64.5%, 41.5%, and 17.4% for 3D-LOTUS++; the paper highlights gains of 10.3 points on novel rigid objects (L2), 10.9 points on novel articulated objects (L3), and 1.6 points on long-horizon tasks (L4). Ablations attribute the gains to multi-view input, history-aware planning, and end-to-end mask generation.

Load-bearing premise

The load-bearing assumption is that Gondola's higher success rates come from the mask-grounded planning model itself, but Gondola replans during execution while the 3D-LOTUS++ baseline plans only once, so part of the improvement could come from the replanning loop.

Editorial extensions

If this is right

  • If the GemBench results transfer, a high-level manipulation planner can be trained largely from simulated, scripted trajectories and still generalize to unseen objects and placements without per-task retraining.
  • The reported ablations imply that multi-view masks are a more reliable interface for a motion planner than textual bounding boxes, since the box-based variant loses grounding quality and format stability.
  • History-aware planning is a double-edged mechanism: it improves coherence on novel objects, and pseudo long-horizon training is what recovers long-horizon performance once history is present.
  • Action-chunk size trades off frequent closed-loop correction against consistency for long-horizon tasks, so deployment should choose chunk size per task rather than globally.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that part of the margin over 3D-LOTUS++ may come from Gondola's closed-loop replanning loop rather than from mask grounding alone; a controlled comparison with equal replanning frequency would separate the two.
  • The pseudo long-horizon data concatenates two training tasks with abrupt scene changes, so the reported L4 gains are likely a lower bound on what could be achieved with true continuous long-horizon demonstrations; a natural next experiment is training on collected long-horizon episodes.
  • The same mask-grounded plan interface should be testable with other dense-grounding vision-language backbones and other simulators; the paper's design makes the representation, not the specific backbone, the load-bearing idea.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Gondola, a vision-language model for grounded robotic task planning that takes multi-view images and text history as input and outputs interleaved action text and per-view segmentation masks for target objects and locations. The model is trained on three synthetic RLBench datasets: robot grounded planning, multi-view referring expressions, and pseudo long-horizon tasks built by concatenating short sequences. The authors evaluate Gondola both offline on grounded planning metrics (action/object accuracy and mask IoU) and online by integrating it with the 3D-LOTUS++ motion planning policy on the GemBench benchmark, reporting improvements over 3D-LOTUS++ across the four generalization levels. The paper also includes ablations on mask versus box outputs, multi-view input, history plans, training data composition, and action chunk size, plus a small real-robot study.

Significance. If the main claims hold, Gondola makes a useful contribution by demonstrating that multi-view grounded planning with segmentation masks can improve generalization in simulated robotic manipulation, and the proposed pseudo long-horizon training data is a practical technique for improving long-horizon planning. The paper is generally careful in its ablations, uses a shared low-level policy for the main comparison, and is unusually honest in reporting failure cases and limitations (Section 6, Appendix B, Appendix C). However, the headline comparison to 3D-LOTUS++ is not fully controlled because the two systems differ in the execution loop, not only in the planning model, and the action chunk size is chosen per level without a clear protocol. The central claim is defensible but needs additional experiments to support the attribution of the gains to the model itself.

major comments (4)
  1. [§4.2, Table 4] The comparison against 3D-LOTUS++ is not controlled for the execution loop. Gondola replans iteratively with action chunk size 5, while 3D-LOTUS++ performs task planning only once and then executes the plan, as stated in §4.2. The reported gains of 10.3%, 10.9%, and 1.6% on L2, L3, and L4 therefore conflate the grounded planning model with the feedback loop that enables replanning and correction. Please add an ablation that gives 3D-LOTUS++ the same iterative replanning mechanism, or that runs Gondola in a single-plan mode, so that the improvement can be attributed to the planning model rather than to replanning.
  2. [Table 3, Appendix B] The action chunk size is a free parameter with large and level-dependent effects: AC=1 outperforms AC=5 on L1 and L2 (90.8 vs 87.3, 78.2 vs 74.8) but underperforms on L3 and L4 (49.5 vs 52.4, 14.9 vs 19.0). The paper reports AC=5 in Table 4 without stating whether this configuration was selected on the validation split or on the test results. The average success rate over the four levels is nearly identical for AC=5 and AC=1 (58.4% vs 58.4%), so the claim of superiority across all levels depends on this per-level choice. Please specify the AC selection protocol and, if possible, report the baseline under both AC configurations.
  3. [§3.2 vs §4.1] The offline grounded planning evaluation uses the same annotation pipeline—RLBench semantic labels plus manual plan decompositions—that generates the training data. Since the paper itself notes in §3.2 that RLBench semantic labels 'contain noises and ambiguities' and require an automatic preprocessing pipeline, the high offline accuracies may partly reflect label artifacts shared between training and evaluation rather than transferable planning ability. Please report a small human-annotated subset or an alternative annotation source to validate the offline metrics, or explicitly discuss this risk in the limitations.
  4. [Table 4] The L4 improvement over 3D-LOTUS++ is small: 19.0±1.0 vs 17.4±0.4, a difference of 1.6 percentage points. Given the reported standard deviations across the five runs, the paper should provide a significance test or a per-task breakdown (already partially in Table 8) to support the claim that Gondola outperforms the baseline on long-horizon tasks. Without this, the L4 component of the 'all four levels' claim is not firmly established.
minor comments (5)
  1. [§4.3, Table 1] The discussion order in the text (boxes vs masks, multi-view, history) does not match the row order in Table 1; please reorder either the table rows or the paragraphs so the reader can follow the ablations more easily.
  2. [§4.1] The offline metrics 'Act' and 'Obj' are defined as exact text match, but it is unclear whether minor formatting differences, synonyms, or color-name variants (e.g., 'navy bulb' vs 'light bulb') are counted as correct; please clarify the matching rule, particularly since the box-based variant is said to suffer from format errors.
  3. [Table 3] The column header '3D filter' is not defined in the caption; the text later explains that it refers to DBSCAN point-cloud filtering, but this should be stated directly in the table caption.
  4. [§1 vs §4.4] The introduction says Gondola outperforms the state-of-the-art LLM-based method 'by absolute 10% on average,' while the per-level gains reported in §4.4 are 10.3%, 10.9%, and 1.6%; please specify the averaging procedure (which levels are included) so the claim is reproducible.
  5. [Appendix C] The real-robot results show lower performance than simulation, and the paper correctly attributes this partly to limited real data and lower multi-view consistency; consider adding a short discussion of how the proposed pseudo long-horizon data or mask-based grounding could be adapted to reduce this sim-to-real gap in future work.

Circularity Check

0 steps flagged · score 1.0 of 10

No definitional circularity: held-out GemBench evaluation and external components keep the derivation independent; the self-cited benchmark and baseline are comparisons, not inputs that force the result.

full rationale

The paper's central claim is an empirical system comparison, not a derivation that reduces to its inputs. Gondola is trained on the GemBench training split and evaluated on held-out GemBench validation/test splits with L2-L4 tasks excluded from training ('To ensure a fair evaluation on generalization, tasks from L2 to L4 are excluded during training'), so the reported success rates and offline planning metrics are supervised evaluations against unseen data. No fitted parameter is later renamed as a prediction, and no equation in Section 3 defines the reported outcome in terms of the training objective. The main self-citations are to [14] (GemBench and 3D-LOTUS++): using one's own benchmark and baseline is legitimate and falsifiable, and the cited prior work supplies the evaluation protocol and motion policy rather than an unverified uniqueness theorem or ansatz that forces Gondola's design. The execution-protocol difference (Gondola replans while 3D-LOTUS++ plans once, Section 4.2) is a real attribution confound for the headline gains, and the AC=5 choice is partly selected from Table 3, but this is a correctness/fairness concern rather than circularity, because the comparison is still measured against independent held-out episodes. The limitations section candidly reports data-scarcity and real-world grounding weaknesses, which further indicates the evaluation is not constructed to guarantee the claimed outcome. No specific reduction of the form 'Eq. X = Eq. Y by construction' or 'fitted parameter renamed as prediction' can be exhibited, so no circular step is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central results rest mostly on the quality of RLBench-derived annotations, the assumption that the reused 3D motion policy executes the plans faithfully, and the chosen evaluation configuration (action chunk size, number of views). No new physical entities are introduced. The ledger lists the hand-tuned constants and domain assumptions that the evaluation depends on.

free parameters (5)
  • action_chunk_size = 5
    Main comparison (Table 4) uses AC=5; with AC=1, Gondola scores 14.9 on L4, below 3D-LOTUS++'s 17.4, so the 'all four levels' claim depends on this choice.
  • number_of_camera_views = 4
    Standard GemBench camera setup; the method is evaluated with K=4, and single-view ablations perform worse.
  • lora_rank = 128
    LoRA rank for parameter-efficient fine-tuning; chosen without reported sweep.
  • learning_rate = 2e-5
    AdamW learning rate for all trainable parameters; chosen without reported sweep.
  • color_names = 20 predefined colors
    Color list used to prepend color names to RLBench object labels (Appendix A); affects reference expressions and object grounding.
assumptions (4)
  • domain assumption RLBench scripted trajectories and semantic labels provide reliable ground-truth plans and masks
    Used to auto-generate training tuples and offline evaluation annotations (Section 3.2, Appendix A); if labels are noisy, the training signal and offline metrics are biased.
  • domain assumption The 3D-LOTUS++ motion planning policy can execute any plan output by Gondola when given the four-class point cloud
    Task success is measured end-to-end through this policy; its limitations (contact-rich tasks, partial views) are responsible for many failures in Appendix B, so the motion policy is load-bearing but not part of the contribution.
  • ad hoc to paper Pseudo long-horizon tasks formed by concatenating two short sequences provide a useful training distribution despite abrupt scene transitions
    Proposed in Section 3.2; authors acknowledge the scenes are discontinuous, and the benefit is empirical (Table 2).
  • domain assumption Exact text matching of action and object names is a valid offline proxy for planning quality
    Used in Section 4.1 for grounded planning evaluation; it does not capture whether the plan is executable or correct in context.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation." pith.science (2026). https://pith.science/paper/TS7NID7T

@misc{pith2026250611261,
  author       = {Pith},
  title        = {Pith review of: Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TS7NID7T}},
  note         = {Machine review of arXiv:2506.11261}
}
read the original abstract

Robotic manipulation faces a significant challenge in generalizing across unseen objects, environments and tasks specified by diverse language instructions. To improve generalization capabilities, recent research has incorporated large language models (LLMs) for planning and action execution. While promising, these methods often fall short in generating grounded plans in visual environments. Although efforts have been made to perform visual instructional tuning on LLMs for robotic manipulation, existing methods are typically constrained by single-view image input and struggle with precise object grounding. In this work, we introduce Gondola, a novel grounded vision-language planning model based on LLMs for generalizable robotic manipulation. Gondola takes multi-view images and history plans to produce the next action plan with interleaved texts and segmentation masks of target objects and locations. To support the training of Gondola, we construct three types of datasets using the RLBench simulator, namely robot grounded planning, multi-view referring expression and pseudo long-horizon task datasets. Gondola outperforms the state-of-the-art LLM-based method across all four generalization levels of the GemBench dataset, including novel placements, rigid objects, articulated objects and long-horizon tasks.

Figures

Figures reproduced from arXiv: 2506.11261 by the authors.

Figure 1
Figure 1. Comparison of vision-language models for high-level planning in robotic manipulation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: Gondola model architecture, consisting of a shared visual encoder for multi-view [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Three types of datasets are constructed for model training: (1) robot grounded planning, (2) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: A failure example of the ‘PutInCupboard’ task using Gondola (AC=5). The instruction is [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: A failure example of the ‘SlideBlock’ task using Gondola (AC=5). The instruction is ‘slide [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Our setup includes three Re￾alSense D435 cameras and a UR5 robotics arm equipped with a RG6 gripper. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Successful examples of Gondola in unseen tasks with real robot. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Failure cases of Gondola in unseen tasks with real robot. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 32 canonical work pages

  1. [1]

    Akkaya, M

    I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand.arXiv preprint arXiv:1910.07113, 2019

    arXiv 1910

  2. [2]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.arXiv:2303.04137, 2023

    arXiv 2023

  3. [3]

    Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.arXiv preprint arXiv:2401.02117, 2024

    arXiv 2024

  4. [4]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv:2212.06817, 2022

    arXiv 2022

  5. [5]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    arXiv 2023

  6. [6]

    Driess, F

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    arXiv 2023

  7. [7]

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    arXiv 2024

  8. [8]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    arXiv 2024

Show all 63 references
  1. [9]

    Vuong, S

    Q. Vuong, S. Levine, H. R. Walke, K. Pertsch, A. Singh, R. Doshi, C. Xu, J. Luo, L. Tan, D. Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. InCoRL, 2023

    2023

  2. [10]

    Khazatsky, K

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. InRSS 2024 Workshop: Data Generation for Robotics. 9

    2024

  3. [11]

    AgiBot-World-Contributors, Q. Bu, J. Cai, L. Chen, X. Cui, Y . Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang, S. Jiang, Y . Jiang, C. Jing, H. Li, J. Li, C. Liu, Y . Liu, Y . Lu, J. Luo, P. Luo, Y . Mu, Y . Niu, Y . Pan, J. Pang, Y . Qiao, G. Ren, C. Ruan, J. Shan, Y . Shen, C...

    2025 arXiv

  4. [12]

    O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. CALVIN: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks.IEEE RA-L, 2022

    2022

  5. [13]

    Pumacay, I

    W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. The colosseum: A bench- mark for evaluating generalization for robotic manipulation.arXiv preprint arXiv:2402.08191, 2024

    2024 arXiv

  6. [14]

    Garcia, S

    R. Garcia, S. Chen, and C. Schmid. Towards generalizable vision-language robotic manipulation: A benchmark and LLM-guided 3D policy.ICRA, 2025

    2025

  7. [15]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. InICRA, 2023

    2023

  8. [16]

    Huang, C

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv:2307.05973, 2023

    2023 arXiv

  9. [17]

    Bjorck, F

    J. Bjorck, F. Casta˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    2025 arXiv

  10. [18]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/ llama3/blob/main/MODEL_CARD.md

    2024

  11. [20]

    Huang, P

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. InICML, 2022

    2022

  12. [21]

    Brohan, Y

    A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. InCoRL, 2023

    2023

  13. [22]

    Michał, C

    Z. Michał, C. William, P. Karl, M. Oier, F. Chelsea, and L. Sergey. Robotic control via embodied chain-of-thought reasoning. InCORL, 2024

    2024

  14. [23]

    X. Li, C. Mata, J. Park, K. Kahatapitiya, Y . S. Jang, J. Shang, K. Ranasinghe, R. Burgert, M. Cai, Y . J. Lee, et al. Llara: Supercharging robot learning data for vision-language policy.arXiv preprint arXiv:2406.20095, 2024

    2024 arXiv

  15. [24]

    W. Yuan, J. Duan, V . Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    2024 arXiv

  16. [25]

    H. Yuan, X. Li, T. Zhang, Z. Huang, S. Xu, S. Ji, Y . Tong, L. Qi, J. Feng, and M.-H. Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025

    2025 arXiv

  17. [26]

    James, Z

    S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment.IEEE RA-L, 2020

    2020

  18. [27]

    L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations.IJRR, 2021. 10

    2021

  19. [28]

    Lynch, A

    C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time.IEEE RA-L, 2023

    2023

  20. [29]

    Stepputtis, J

    S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor. Language- conditioned imitation learning for robot manipulation tasks.NeurIPS, 2020

    2020

  21. [30]

    Kalashnikov, A

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V . Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. InCoRL, 2018

    2018

  22. [31]

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. InCoRL, 2022

    2022

  23. [32]

    Guhur, S

    P.-L. Guhur, S. Chen, R. Garcia Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. InCoRL, 2023

    2023

  24. [33]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. InCoRL, 2023

    2023

  25. [34]

    Goyal, J

    A. Goyal, J. Xu, Y . Guo, V . Blukis, Y .-W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. InCoRL, 2023

    2023

  26. [35]

    S. Chen, R. Garcia, C. Schmid, and I. Laptev. Polarnet: 3d point clouds for language-guided robotic manipulation. InCoRL, 2023

    2023

  27. [36]

    S. Chen, R. Garcia, I. Laptev, and C. Schmid. Sugar: Pre-training 3d visual representations for robotics.CVPR, 2024

    2024

  28. [37]

    Gervet, Z

    T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. InCoRL, 2023

    2023

  29. [38]

    T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations.arXiv:2402.10885, 2024

    2024 arXiv

  30. [39]

    Goyal, V

    A. Goyal, V . Blukis, J. Xu, Y . Guo, Y .-W. Chao, and D. Fox. Rvt2: Learning precise manipula- tion from few demonstrations. InRSS, 2024

    2024

  31. [40]

    Tziafas and H

    G. Tziafas and H. Kasaei. Towards open-world grasping with large vision-language models. arXiv preprint arXiv:2406.18722, 2024

    2024 arXiv

  32. [41]

    James, K

    S. James, K. Wada, T. Laidlow, and A. J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. InCVPR, 2022

    2022

  33. [42]

    Chisari, N

    E. Chisari, N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada. Learning robotic manipulation policies from point clouds with conditional flow matching.arXiv preprint arXiv:2409.07343, 2024

    2024 arXiv

  34. [43]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021

    2021

  35. [44]

    T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, D. M, J. Peralta, B. Ichter, K. Hausman, and F. Xia. Scaling robot learning with semantically imagined experience. arXiv:2302.11550, 2023

    2023 arXiv

  36. [45]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll´ar, and R. Girshick. Segment anything.arXiv:2304.02643, 2023

    2023 arXiv

  37. [46]

    Y . Hu, F. Lin, T. Zhang, L. Yi, and Y . Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning.arXiv:2311.17842, 2023. 11

    2023 arXiv

  38. [47]

    G. OpenAI. 4v (ision) system card.preprint, 2023

    2023

  39. [48]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.NeurIPS, 2024

    2024

  40. [49]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning. InCVPR, 2024

    2024

  41. [50]

    Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  42. [51]

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  43. [52]

    J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V . Chandra, Y . Xiong, and M. Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.arXiv preprint arXiv:2310.09478, 2023

    arXiv

  44. [53]

    H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y . Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    2023 arXiv

  45. [54]

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y . Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al. Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  46. [55]

    J. Liu, H. Ding, Z. Cai, Y . Zhang, R. K. Satzoda, V . Mahadevan, and R. Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. InCVPR, 2023

    2023

  47. [56]

    C. Ma, Y . Jiang, J. Wu, Z. Yuan, and X. Qi. Groma: Localized visual tokenization for grounding multimodal large language models. InECCV, 2025

    2025

  48. [57]

    Zhang, Z

    Y . Zhang, Z. Ma, X. Gao, S. Shakiah, Q. Gao, and J. Chai. Groundhog: Grounding large language models to holistic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14227–14238, 2024

    2024

  49. [58]

    X. Lai, Z. Tian, Y . Chen, Y . Li, Y . Yuan, S. Liu, and J. Jia. Lisa: Reasoning segmentation via large language model. InCVPR, 2024

    2024

  50. [59]

    Zhang, H

    H. Zhang, H. Li, F. Li, T. Ren, X. Zou, S. Liu, S. Huang, J. Gao, C. Li, J. Yang, et al. Llava- grounding: Grounded visual chat with large multimodal models. InECCV. Springer, 2025

    2025

  51. [60]

    Rasheed, M

    H. Rasheed, M. Maaz, S. Shaji, A. Shaker, S. Khan, H. Cholakkal, R. M. Anwer, E. Xing, M.-H. Yang, and F. S. Khan. Glamm: Pixel grounding large multimodal model. InCVPR, 2024

    2024

  52. [61]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2024

  53. [62]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  54. [63]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  55. [64]

    Rasley, S

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InACM SIGKDD, 2020. 12 A Data Construction in RLBench We detail the label construction from RLBench in Section 3.2. RLBench ...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.