Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning

Bowen Yang; Long Zeng; Siyu Chen; Tongtong Cao; Xinyu Shao; Xiu Li; Yajun Gao; Yanzhe Tang; Yuxuan Hu

arxiv: 2606.25360 · v1 · pith:357YE3EX · submitted 2026-06-24 · cs.RO

Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning

Yanzhe Tang , Xinyu Shao , Yuxuan Hu , Siyu Chen , Bowen Yang , Yajun Gao , Tongtong Cao , Xiu Li

show 1 more author

Long Zeng

This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-25 21:21 UTCgrok-4.3pith:357YE3EX record.json open to challenge →

classification cs.RO

keywords imitation learningvision language actionspatial visual promptsrobotic manipulationlanguage conditioned tasksdecoupled architecturezero-shot groundingdata efficient learning

0 comments

The pith

Decoupling semantics from geometric grounding via spatial visual prompts improves language-conditioned imitation learning with limited data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to show that end-to-end vision-language-action models suffer from an alignment problem when semantic understanding and spatial control are mixed together, especially with scarce demonstration data. It proposes separating the spatial grounding step by using vision-language models to create zero-shot geometric masks that serve as explicit spatial visual prompts. These prompts are then fused directly into a continuous action generator. This separation is claimed to provide stable spatial guidance and better performance on ambiguous tasks. A reader would care if this means robots can learn to follow vague language instructions more reliably without massive datasets.

Core claim

The central claim is that by explicitly extracting spatial visual grounding from the action generation loop using zero-shot geometric masks parsed from instructions by vision-language foundation models and injecting them as Spatial Visual Prompts into a lightweight feature-level fusion mechanism, the architecture overcomes the alignment bottleneck and achieves superior success rates in data-constrained language-conditioned robotic manipulation.

What carries the argument

Spatial Visual Prompts (SVP), which are zero-shot geometric masks from vision-language foundation models that translate language into explicit spatial priors for direct fusion into the action generator.

If this is right

SVP-IL achieves 67.8% success on standard benchmarks while outperforming state-of-the-art VLAs.
Improves average success rates on highly ambiguous language-conditioned tasks from 24.0% to 39.5% when trained on 50 to 100 demonstrations.
The approach ensures highly stable optimization under low-data regimes.
Real-world experiments validate robustness in unstructured physical environments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This separation might allow easier debugging of spatial errors separately from semantic ones in robotic systems.
Future work could test whether similar decoupling benefits other multimodal control tasks beyond manipulation.
The reliance on foundation models for masks suggests potential for rapid adaptation to new environments without retraining the entire system.

Load-bearing premise

The zero-shot geometric masks extracted by vision-language foundation models provide accurate and uncorrupted spatial information that fuses without alignment errors or requiring task-specific fine-tuning.

What would settle it

Running the method with deliberately corrupted or misaligned masks and observing no improvement or worse performance than coupled baselines would falsify the benefit of the decoupling.

Figures

Figures reproduced from arXiv: 2606.25360 by Bowen Yang, Long Zeng, Siyu Chen, Tongtong Cao, Xinyu Shao, Xiu Li, Yajun Gao, Yanzhe Tang, Yuxuan Hu.

**Figure 1.** Figure 1: Architectural comparison between standard endto-end VLAs and our SVP-IL. (a) End-to-end VLA models jointly train the vision-language backbone and action expert, passing information via an implicit latent vector. This blackbox process often leads to entangled semantic and spatial features. (b) In contrast, SVP-IL employs a frozen visionlanguage model to extract explicit geometric masks from instructions.… view at source ↗

**Figure 2.** Figure 2: Overview of the SVP-IL framework. Semantic reasoning is offloaded to a two-stage foundation model pipeline (LLM [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Visualization of the simulation tasks and environmental distributions. The figure illustrates the two distinct visual domains used for expert demonstration collection and policy evaluation. The left column shows the Clean setting with minimalist backgrounds. The right columns display the Randomized setting, which introduces severe distribution shifts including complex textures, dynamic lighting, and dense… view at source ↗

**Figure 4.** Figure 4: Real-world bimanual evaluation setup and diverse tasks on the Aloha-AgileX platform. The top-left panel details the physical setup, where our policy relies exclusively on the head-mounted RGB camera using a 2-frame history as input. The remaining panels showcase three distinct manipulation tasks: Table Setting, Desktop Cleaning, and Bimanual Sorting. Across all scenarios, the workspace features a dense cl… view at source ↗

read the original abstract

While end-to-end Vision-Language-Action (VLA) models show promise in robotic manipulation, their monolithic paradigm inherently couples semantic reasoning and spatial control. This creates a severe alignment bottleneck, limiting precise target disambiguation in data-constrained imitation learning. To overcome this, we propose SVP-IL, a decoupled architecture that explicitly extracts spatial visual grounding from the action generation loop. By leveraging vision-language foundation models, we parse instructions into zero-shot geometric masks, translating language into explicit Spatial Visual Prompts (SVP). These priors are injected into a continuous action generator via a lightweight direct feature-level fusion mechanism. This integration provides explicit and uncorrupted spatial gradient guidance while ensuring highly stable optimization under low-data regimes. Extensive experiments demonstrate that SVP-IL significantly outperforms state-of-the-art VLAs and pure visuomotor baselines. Trained on as few as 50 to 100 demonstrations, SVP-IL improves average success rates on highly ambiguous language-conditioned tasks from 24.0% to 39.5%, achieving 67.8% on standard benchmarks. Real-world robotic experiments further validate its robustness and data efficiency in unstructured physical environments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

SVP-IL decouples VLA alignment by feeding zero-shot VLM geometric masks as spatial prompts into a separate action generator, claiming solid gains with 50-100 demos, but the mask accuracy assumption and thin experimental details are the main things to watch.

read the letter

The main thing to know is that this paper proposes SVP-IL to break the coupling between language semantics and spatial control in imitation learning. It pulls geometric masks from a vision-language model in zero-shot fashion, turns them into explicit spatial visual prompts, and fuses them at the feature level into a continuous action generator. The goal is to give the action side stable spatial guidance without forcing the whole model to learn the alignment from scratch under low data.

What stands out as new is the targeted use of those zero-shot masks for this specific decoupling, plus the lightweight fusion step that avoids retraining the mask generator. The reported numbers show average success on ambiguous tasks rising from 24% to 39.5% and reaching 67.8% on standard benchmarks, with real-robot tests mentioned as further support. That kind of data efficiency is worth noting for anyone dealing with language-conditioned manipulation where demos are expensive.

The approach builds directly on existing VLM and imitation components rather than claiming a full new paradigm, but the modular split and direct injection are a reasonable engineering response to the alignment bottleneck.

The soft spots are concentrated on the central assumption. The gains are attributed to the masks supplying uncorrupted spatial information, yet there is no reported check on mask accuracy, alignment errors, or what happens when the VLM gets the geometry wrong on the actual tasks. The abstract also gives no information on baseline implementations, statistical significance, or failure modes, which leaves the quantitative claims hard to evaluate from the summary alone. If the full methods section does not address these, the data-efficiency story rests on untested priors.

This is for people working on practical language-conditioned robot learning who want modular ways to reduce data needs. A reader already familiar with VLA limitations could extract the fusion idea and test it themselves.

I would send it to peer review. The problem it targets is real, the architecture is straightforward, and referees can check whether the mask assumption and experiments hold up.

Referee Report

2 major / 0 minor

Summary. The paper claims that monolithic VLAs suffer from an alignment bottleneck between semantics and spatial control in data-scarce imitation learning. It proposes SVP-IL, which decouples these by using vision-language foundation models to extract zero-shot geometric masks as Spatial Visual Prompts (SVP). These masks are fused via lightweight feature-level injection into a continuous action generator to supply explicit spatial gradient guidance. With 50-100 demonstrations, SVP-IL is reported to raise average success on ambiguous language-conditioned tasks from 24.0% to 39.5% and reach 67.8% on standard benchmarks, outperforming VLAs and visuomotor baselines; real-world robotic validation is also claimed.

Significance. If the reported gains are reproducible and attributable to the decoupling rather than mask artifacts or baseline weaknesses, the approach could meaningfully improve data efficiency for language-conditioned manipulation by supplying stable spatial priors without VLA-scale alignment training.

major comments (2)

[Abstract] Abstract: the central claim that zero-shot geometric masks supply 'explicit and uncorrupted spatial gradient guidance' without alignment errors or task-specific tuning is load-bearing for the data-efficiency attribution, yet the text provides no mask-accuracy metrics, alignment validation, or failure-case analysis on the target tasks.
[Abstract] Abstract: quantitative results (24.0% → 39.5%, 67.8% benchmark) are stated without any description of experimental controls, baseline re-implementations, trial counts, variance, or statistical tests, preventing assessment of whether the SVP fusion is responsible for the gains.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract. We address each major comment below and will revise the manuscript to improve clarity and support for the central claims.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that zero-shot geometric masks supply 'explicit and uncorrupted spatial gradient guidance' without alignment errors or task-specific tuning is load-bearing for the data-efficiency attribution, yet the text provides no mask-accuracy metrics, alignment validation, or failure-case analysis on the target tasks.

Authors: We agree that the abstract would benefit from additional supporting detail on this point. Because the masks are produced zero-shot by off-the-shelf vision-language models, task-specific quantitative accuracy metrics would require new ground-truth annotations that were not collected; however, we will revise the abstract to reference the documented spatial-grounding performance of the underlying foundation models and to summarize the failure-case analysis already present in Section 5.3 of the full manuscript. revision: yes
Referee: [Abstract] Abstract: quantitative results (24.0% → 39.5%, 67.8% benchmark) are stated without any description of experimental controls, baseline re-implementations, trial counts, variance, or statistical tests, preventing assessment of whether the SVP fusion is responsible for the gains.

Authors: The abstract is a concise summary; the full experimental protocol—including baseline re-implementations, 10 evaluation trials per task, standard-deviation reporting, and paired statistical tests—is detailed in Section 4. We will revise the abstract to include a short clause referencing the evaluation protocol and directing readers to the Experiments section for the complete controls and statistics. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical claims rest on experiments

full rationale

The paper presents an architectural proposal (SVP-IL) and reports experimental results on success rates with 50-100 demonstrations. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text. The central claims are supported by benchmark comparisons rather than reducing to self-definitional inputs or ansatzes. This matches the default expectation for non-circular empirical work.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; the approach relies on standard vision-language foundation models and imitation learning assumptions without detailing new postulates.

pith-pipeline@v0.9.1-grok · 5762 in / 1014 out tokens · 23202 ms · 2026-06-25T21:21:45.525340+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

22 extracted references · 6 canonical work pages · 2 internal anchors

[1]

π 0: A vision-language-action flow model for general robot control,

K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichteret al., “π 0: A vision-language-action flow model for general robot control,” inProceedings of Robotics: Science and Systems (RSS), 2025

2025
[2]

OpenVLA: An open-source vision-language-action model,

M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketiet al., “OpenVLA: An open-source vision-language-action model,” inProceedings of the Conference on Robot Learning (CoRL), 2024

2024
[3]

SAM 3: Segment Anything with Concepts

N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huanget al., “SAM 3: Segment anything with concepts,”arXiv preprint arXiv:2511.16719, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[4]

ALVINN: An autonomous land vehicle in a neural network,

D. A. Pomerleau, “ALVINN: An autonomous land vehicle in a neural network,”Advances in Neural Information Processing Systems (NeurIPS), vol. 1, 1988

1988
[5]

Robotic manipulation via imitation learning: Taxonomy, evolution, benchmark, and challenges,

Z. Li, A. Chapin, E. Xiang, R. Yang, B. Machado, N. Lei, E. Dellan- drea, D. Huang, and L. Chen, “Robotic manipulation via imitation learning: Taxonomy, evolution, benchmark, and challenges,”arXiv preprint arXiv:2508.17449, 2025

work page arXiv 2025
[6]

Learning fine-grained bimanual manipulation with low-cost hardware,

T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” inProceedings of Robotics: Science and Systems (RSS), 2023

2023
[7]

Diffusion policy: Visuomotor policy learning via action diffusion,

C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” inProceedings of Robotics: Science and Systems (RSS), 2023

2023
[8]

FiLM: Visual reasoning with a general conditioning layer,

E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “FiLM: Visual reasoning with a general conditioning layer,” inPro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018

2018
[9]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inProceedings of the International Conference on Machine Learning (ICML), 2021

2021
[10]

RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation

T. Chen, Z. Chen, B. Chen, Z. Cai, Y . Liu, Z. Li, Q. Liang, X. Lin, Y . Ge, Z. Guet al., “Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation,”arXiv preprint arXiv:2506.18088, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[11]

3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations,

Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations,” inProceedings of Robotics: Science and Systems (RSS), 2024

2024
[12]

RISE: 3D perception makes real-world robot imitation simple and effective,

H. Fang, C. Wang, H.-S. Fang, and C. Lu, “RISE: 3D perception makes real-world robot imitation simple and effective,” inProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024

2024
[13]

RT-2: Vision-language-action models transfer web knowledge to robotic control,

B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahidet al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” inProceedings of the Conference on Robot Learning (CoRL), 2023, pp. 2165–2183

2023
[14]

Dense object nets: Learn- ing dense visual object descriptors by and for robotic manipulation,

P. R. Florence, L. Manuelli, and R. Tedrake, “Dense object nets: Learn- ing dense visual object descriptors by and for robotic manipulation,” inProceedings of the Conference on Robot Learning (CoRL), 2018, pp. 373–385

2018
[15]

VIMA: General robot manipu- lation with multimodal prompts,

Y . Jiang, A. Gupta, Z. Zhang, G. Wang, Y . Dou, Y . Chen, L. Fei-Fei, A. Anandkumar, Y . Zhu, and L. Fan, “VIMA: General robot manipu- lation with multimodal prompts,” inProceedings of the International Conference on Machine Learning (ICML), 2023

2023
[16]

MOKA: Open-vocabulary robotic manipulation through mark-based visual prompting,

F. Liu, K. Fang, P. Abbeel, and S. Levine, “MOKA: Open-vocabulary robotic manipulation through mark-based visual prompting,” inPro- ceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024

2024
[17]

ReKep: Spatio- temporal relational keypoint constraints for generalizable robotic ma- nipulation,

W. Huang, G. Wang, Y . Li, L. Fei-Fei, and Y . Zhu, “ReKep: Spatio- temporal relational keypoint constraints for generalizable robotic ma- nipulation,” inProceedings of the Conference on Robot Learning (CoRL), 2024

2024
[18]

Bring my cup! personalizing vision-language-action models with visual attentive prompting,

S. Lee, S. Mo, and W.-S. Han, “Bring my cup! personalizing vision-language-action models with visual attentive prompting,”arXiv preprint arXiv:2412.20014, 2024

work page arXiv 2024
[19]

Spatial forcing: Implicit spatial representation alignment for vision-language-action model,

F. Li, W. Song, H. Zhao, J. Wang, P. Ding, D. Wang, L. Zeng, and H. Li, “Spatial forcing: Implicit spatial representation alignment for vision-language-action model,”arXiv preprint arXiv:2510.12276, 2025

work page arXiv 2025
[20]

More than a point: Capturing uncertainty with adaptive affordance heatmaps for spatial grounding in robotic tasks,

X. Shao, Y . Tang, P. Xie, K. Zhou, Y . Zhuang, X. Quan, J. Hao, L. Zeng, and X. Li, “More than a point: Capturing uncertainty with adaptive affordance heatmaps for spatial grounding in robotic tasks,” arXiv preprint arXiv:2510.10912, 2025

work page arXiv 2025
[21]

SAM2Act: Integrating visual foundation model with a memory architecture for robotic manipulation,

H. Fang, M. Grotz, W. Pumacay, Y . R. Wang, D. Fox, R. Krishna, and J. Duan, “SAM2Act: Integrating visual foundation model with a memory architecture for robotic manipulation,” inProceedings of the 42nd International Conference on Machine Learning (ICML), 2025, pp. 15 925–15 942

2025
[22]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

2016

[1] [1]

π 0: A vision-language-action flow model for general robot control,

K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichteret al., “π 0: A vision-language-action flow model for general robot control,” inProceedings of Robotics: Science and Systems (RSS), 2025

2025

[2] [2]

OpenVLA: An open-source vision-language-action model,

M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketiet al., “OpenVLA: An open-source vision-language-action model,” inProceedings of the Conference on Robot Learning (CoRL), 2024

2024

[3] [3]

SAM 3: Segment Anything with Concepts

N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huanget al., “SAM 3: Segment anything with concepts,”arXiv preprint arXiv:2511.16719, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025

[4] [4]

ALVINN: An autonomous land vehicle in a neural network,

D. A. Pomerleau, “ALVINN: An autonomous land vehicle in a neural network,”Advances in Neural Information Processing Systems (NeurIPS), vol. 1, 1988

1988

[5] [5]

Robotic manipulation via imitation learning: Taxonomy, evolution, benchmark, and challenges,

Z. Li, A. Chapin, E. Xiang, R. Yang, B. Machado, N. Lei, E. Dellan- drea, D. Huang, and L. Chen, “Robotic manipulation via imitation learning: Taxonomy, evolution, benchmark, and challenges,”arXiv preprint arXiv:2508.17449, 2025

work page arXiv 2025

[6] [6]

Learning fine-grained bimanual manipulation with low-cost hardware,

T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” inProceedings of Robotics: Science and Systems (RSS), 2023

2023

[7] [7]

Diffusion policy: Visuomotor policy learning via action diffusion,

C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” inProceedings of Robotics: Science and Systems (RSS), 2023

2023

[8] [8]

FiLM: Visual reasoning with a general conditioning layer,

E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “FiLM: Visual reasoning with a general conditioning layer,” inPro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018

2018

[9] [9]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inProceedings of the International Conference on Machine Learning (ICML), 2021

2021

[10] [10]

RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation

T. Chen, Z. Chen, B. Chen, Z. Cai, Y . Liu, Z. Li, Q. Liang, X. Lin, Y . Ge, Z. Guet al., “Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation,”arXiv preprint arXiv:2506.18088, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025

[11] [11]

3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations,

Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations,” inProceedings of Robotics: Science and Systems (RSS), 2024

2024

[12] [12]

RISE: 3D perception makes real-world robot imitation simple and effective,

H. Fang, C. Wang, H.-S. Fang, and C. Lu, “RISE: 3D perception makes real-world robot imitation simple and effective,” inProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024

2024

[13] [13]

RT-2: Vision-language-action models transfer web knowledge to robotic control,

B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahidet al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” inProceedings of the Conference on Robot Learning (CoRL), 2023, pp. 2165–2183

2023

[14] [14]

Dense object nets: Learn- ing dense visual object descriptors by and for robotic manipulation,

P. R. Florence, L. Manuelli, and R. Tedrake, “Dense object nets: Learn- ing dense visual object descriptors by and for robotic manipulation,” inProceedings of the Conference on Robot Learning (CoRL), 2018, pp. 373–385

2018

[15] [15]

VIMA: General robot manipu- lation with multimodal prompts,

Y . Jiang, A. Gupta, Z. Zhang, G. Wang, Y . Dou, Y . Chen, L. Fei-Fei, A. Anandkumar, Y . Zhu, and L. Fan, “VIMA: General robot manipu- lation with multimodal prompts,” inProceedings of the International Conference on Machine Learning (ICML), 2023

2023

[16] [16]

MOKA: Open-vocabulary robotic manipulation through mark-based visual prompting,

F. Liu, K. Fang, P. Abbeel, and S. Levine, “MOKA: Open-vocabulary robotic manipulation through mark-based visual prompting,” inPro- ceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024

2024

[17] [17]

ReKep: Spatio- temporal relational keypoint constraints for generalizable robotic ma- nipulation,

W. Huang, G. Wang, Y . Li, L. Fei-Fei, and Y . Zhu, “ReKep: Spatio- temporal relational keypoint constraints for generalizable robotic ma- nipulation,” inProceedings of the Conference on Robot Learning (CoRL), 2024

2024

[18] [18]

Bring my cup! personalizing vision-language-action models with visual attentive prompting,

S. Lee, S. Mo, and W.-S. Han, “Bring my cup! personalizing vision-language-action models with visual attentive prompting,”arXiv preprint arXiv:2412.20014, 2024

work page arXiv 2024

[19] [19]

Spatial forcing: Implicit spatial representation alignment for vision-language-action model,

F. Li, W. Song, H. Zhao, J. Wang, P. Ding, D. Wang, L. Zeng, and H. Li, “Spatial forcing: Implicit spatial representation alignment for vision-language-action model,”arXiv preprint arXiv:2510.12276, 2025

work page arXiv 2025

[20] [20]

More than a point: Capturing uncertainty with adaptive affordance heatmaps for spatial grounding in robotic tasks,

X. Shao, Y . Tang, P. Xie, K. Zhou, Y . Zhuang, X. Quan, J. Hao, L. Zeng, and X. Li, “More than a point: Capturing uncertainty with adaptive affordance heatmaps for spatial grounding in robotic tasks,” arXiv preprint arXiv:2510.10912, 2025

work page arXiv 2025

[21] [21]

SAM2Act: Integrating visual foundation model with a memory architecture for robotic manipulation,

H. Fang, M. Grotz, W. Pumacay, Y . R. Wang, D. Fox, R. Krishna, and J. Duan, “SAM2Act: Integrating visual foundation model with a memory architecture for robotic manipulation,” inProceedings of the 42nd International Conference on Machine Learning (ICML), 2025, pp. 15 925–15 942

2025

[22] [22]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

2016