pith. sign in

arxiv: 2606.08169 · v1 · pith:Y75YURTQnew · submitted 2026-06-06 · 💻 cs.RO · cs.AI· cs.CL· cs.HC· cs.LG

CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning

Markus Knauer , Valentin Gieraths , Tai Mai , Samuel Bustamante , Alin Albu-Sch\"affer , Freek Stulp , Jo\~ao Silv\'erio This is my paper

Pith reviewed 2026-06-27 19:19 UTC · model grok-4.3

classification 💻 cs.RO cs.AIcs.CLcs.HCcs.LG
keywords task-parameterized learningvision-language modelsskill compositionrobot manipulationactive learningkernelized movement primitivesnatural language commandsmodular architecture
0
0 comments X

The pith

Pretrained vision-language models combined with task-parameterized movement primitives enable language-driven skill selection, composition, and active learning on robots without fine-tuning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a modular architecture that pairs task-parameterized kernelized movement primitives with pretrained vision-language models. Skills are learned from two to five kinesthetic demonstrations while the VLM generates schemas that describe each skill's parameters and preconditions. During execution the VLM interprets natural-language commands to select skills, bind parameters, and produce new behaviors through covariance-weighted composition of the primitives. When no existing skill or composition meets the command, the system identifies the gap and requests targeted new demonstrations. The approach is validated on a 7-DoF manipulator across scenarios that require skill selection, composition, and active learning, with reported success rates of 73.3 percent to 100 percent.

Core claim

A modular architecture combines TP-KMPs with pretrained VLMs so that skills acquired from few kinesthetic demonstrations receive language-grounded schemas; at runtime the VLM selects skills, reasons about parameter bindings, and forms novel behaviors by covariance-weighted composition, while also detecting capability gaps and requesting active demonstrations, all without any fine-tuning of the models.

What carries the argument

Covariance-weighted composition of TP-KMPs, driven by VLM-generated skill schemas that encode parameters and preconditions for selection and binding.

Load-bearing premise

The pretrained VLM can reliably produce accurate skill schemas and correctly interpret commands to select and bind skills without hallucination or systematic error.

What would settle it

A command that causes the VLM to select the wrong skill or bind an incorrect parameter, resulting in execution failure or unsafe motion on the 7-DoF manipulator.

Figures

Figures reproduced from arXiv: 2606.08169 by Alin Albu-Sch\"affer, Freek Stulp, Jo\~ao Silv\'erio, Markus Knauer, Samuel Bustamante, Tai Mai, Valentin Gieraths.

Figure 1
Figure 1. Figure 1: Left: execution pipeline illustrated for the command “Insert the bearing ring”: skill matching finds no match (1), composition fails because an insert skill is missing (2), the system acquires the missing skill via new demonstrations (3), composition now succeeds and creates a fused Pick & Insert skill (4), which is then selected (5). Selected skills are parameterized with detected object poses and execute… view at source ↗
Figure 2
Figure 2. Figure 2: Skill schema creation in the learning phase: the VLM creates a schema as well as chooses relevant objects for the task from image input. The relevant objects, together with their pose estimation from the perception pipeline and the kinesthetic demon￾stration, are used to train a TP-KMP skill. Combined with its schema, the final skill is stored in the skill library. The full TP-KMP formalism is pro￾vided in… view at source ↗
Figure 3
Figure 3. Figure 3: Compatibility constraint examples. (a) Compatible: com￾plementary variance creates non￾overlapping dominant regions. (b)– (c) Incompatible: both KMPs have uniformly high or low variance, preventing skill fusion. Compatibility Constraints. The compatibility constraint is a pre-check that validates whether two local KMPs can be com￾posed. Here, P = 2 denotes the number of local KMPs being composed (one selec… view at source ↗
Figure 4
Figure 4. Figure 4: Object generalization success rates across object pairs. Each cell shows the success rate when the skill [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of the object generalization evaluation: switching to unlearned objects for pick-place and [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples for the pose generalization evaluation: trying different pick- and pour positions. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pose generalization robustness. Comparison of automatic vision-based pose estimation (79.3% suc [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Spatial distribution of evaluated object positions for pose generalization. Blue markers indicate [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Examples of the skill combination robustness evaluation showing different box and measurement [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Trajectory fusion of grasp and insert skills via TP-KMP covariance-weighted composition. Demon [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Spatial distribution of evaluated object positions for skill combination. Blue markers indicate suc [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
read the original abstract

Enabling robots to understand and execute tasks from natural language commands while maintaining data efficiency remains challenging. Foundation models such as vision-language-action (VLA) and vision-language models (VLMs) provide intuitive interaction channels but require extensive data; task-parameterized imitation learning achieves data efficiency but lacks natural language grounding. This work bridges this gap through a modular architecture combining task-parameterized kernelized movement primitives (TP-KMPs) with pretrained VLMs. During learning, skills are acquired from 2 to 5 kinesthetic demonstrations, and the VLM generates skill schemas describing each skill's parameters and preconditions. During execution, the VLM interprets commands to select skills, reason about parameter bindings, and create novel behaviors through covariance-weighted composition. When no skill or composition suffices, the system identifies capability gaps and requests targeted demonstrations, all without fine-tuning. Validation on a 7-DoF manipulator shows success rates of 73.3%-100% in scenarios requiring skill selection, composition, and active learning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces CLASP, a modular architecture integrating task-parameterized kernelized movement primitives (TP-KMPs) with pretrained vision-language models (VLMs) for language-driven robot skill selection, composition, and active learning. Skills are learned from 2-5 kinesthetic demonstrations, with the VLM generating schemas for parameters and preconditions. During execution, the VLM handles command interpretation, skill selection, parameter binding, covariance-weighted composition, and requests for new demos when needed, without fine-tuning. Experiments on a 7-DoF manipulator report success rates between 73.3% and 100% across scenarios involving selection, composition, and active learning.

Significance. If the empirical results hold under scrutiny, the work demonstrates a practical, data-efficient alternative to fine-tuning large vision-language-action models by combining modular imitation learning with off-the-shelf VLMs, enabling skill composition and active learning from natural language while avoiding extensive retraining.

major comments (2)
  1. Abstract: success rates of 73.3%-100% are stated without any description of the number of trials, task definitions, baselines, statistical measures, or failure modes, rendering the central validation claim unevaluable.
  2. Execution phase (as described): the architecture has no fallback or correction for VLM outputs, yet the central claim depends on the VLM reliably producing accurate skill schemas from 2-5 demos and correctly interpreting commands for selection, binding, and composition; no quantitative VLM error analysis or robustness tests are referenced.
minor comments (1)
  1. The abstract would be strengthened by naming the specific VLM and providing at least one concrete example of a skill schema or command interpretation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the clarity of our experimental claims and the need for robustness analysis. We address each major comment below and commit to revisions that strengthen the manuscript without altering its core contributions.

read point-by-point responses

  1. Referee: [—] Abstract: success rates of 73.3%-100% are stated without any description of the number of trials, task definitions, baselines, statistical measures, or failure modes, rendering the central validation claim unevaluable.

    Authors: The abstract serves as a concise summary, while the full experimental protocol—including 15 trials per scenario, explicit task definitions, baseline comparisons, mean success rates with standard deviations, and failure mode analysis—is detailed in Section V. To improve self-containment, we will revise the abstract to briefly reference the number of trials, key metrics, and that results are aggregated over multiple runs. revision: yes

  2. Referee: [—] Execution phase (as described): the architecture has no fallback or correction for VLM outputs, yet the central claim depends on the VLM reliably producing accurate skill schemas from 2-5 demos and correctly interpreting commands for selection, binding, and composition; no quantitative VLM error analysis or robustness tests are referenced.

    Authors: The active learning component functions as a built-in response to insufficient VLM outputs by requesting new demonstrations when no skill or composition matches. We agree that quantitative VLM error analysis is absent and will add a dedicated subsection in the Experiments section reporting observed error rates for schema generation and command interpretation, along with robustness tests across prompt variations. revision: yes

Circularity Check

0 steps flagged

No significant circularity; modular system uses external pretrained components

full rationale

The paper presents an engineering architecture that combines existing TP-KMPs (from prior literature) with off-the-shelf pretrained VLMs for skill schema generation, selection, binding, and covariance-weighted composition. No equations, parameter fits, or first-principles derivations are described whose outputs reduce to the inputs by construction. Empirical success rates (73.3%-100%) are reported from robot experiments rather than any self-referential prediction step. The central premise (reliable VLM behavior) is an external assumption, not a derived quantity internal to the paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; all technical details remain opaque.

pith-pipeline@v0.9.1-grok · 5743 in / 1218 out tokens · 15346 ms · 2026-06-27T19:19:56.084285+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 33 canonical work pages · 3 internal anchors

  1. [1]

    S. Schaal. Is imitation learning the route to humanoid robots?Trends in Cognitive Sciences, 3 (6):233–242, 1999. doi:10.1016/S1364-6613(99)01327-3

    work page doi:10.1016/s1364-6613(99)01327-3 1999

  2. [2]

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration.Robotics and Autonomous Systems, 57(5):469–483, 2009. doi:10.1016/j.robot. 2008.10.024

    work page doi:10.1016/j.robot 2009

  3. [3]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model. InConference on Robot Learning (CoRL), volume 270 ofProceedings of Machine Learning Research, pag...

    2025

  4. [4]

    W. Yuan, J. Duan, V . Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction in robotics. InCon- ference on Robot Learning (CoRL), volume 270 ofProceedings of Machine Learning Re- search, pages 4005–4020. PMLR, 2025. URLhttps://proceedings.mlr.press/v270/ yuan25c.html

    2025

  5. [5]

    Raman, Ankit Shah, and Stefanie Tellex

    A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, et al. Open X-Embodiment: Robotic learn- ing datasets and RT-X models. InIEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903, 2024. doi:10.1109/ICRA57147.2024.10611477

    work page doi:10.1109/icra57147.2024.10611477 2024

  6. [6]

    S. Calinon. A tutorial on task-parameterized movement learning and retrieval.Intelligent Service Robotics, 9(1):1–29, 2016. doi:10.1007/s11370-015-0187-9

    work page doi:10.1007/s11370-015-0187-9 2016

  7. [7]

    Huang, L

    Y . Huang, L. Rozo, J. Silv ´erio, and D. G. Caldwell. Kernelized movement primitives. International Journal of Robotics Research (IJRR), 38(7):833–852, 2019. doi:10.1177/ 0278364919846363

    2019

  8. [8]

    Knauer, A

    M. Knauer, A. Albu-Sch ¨affer, F. Stulp, and J. Silv ´erio. Interactive incremental learning of generalizable skills with local trajectory modulation.IEEE Robotics and Automation Letters (RA-L), 10(4):3398–3405, 2025. doi:10.1109/LRA.2025.3542209

    work page doi:10.1109/lra.2025.3542209 2025

  9. [9]

    Saveriano, F

    M. Saveriano, F. J. Abu-Dakka, A. Kramberger, and L. Peternel. Dynamic movement primi- tives in robotics: A tutorial survey.International Journal of Robotics Research (IJRR), 42(13): 1133–1184, 2023. doi:10.1177/02783649231201196

    work page doi:10.1177/02783649231201196 2023

  10. [10]

    Calinon, D

    S. Calinon, D. Bruno, and D. G. Caldwell. A task-parameterized probabilistic model with minimal intervention control. InIEEE International Conference on Robotics and Automation (ICRA), pages 3339–3344, 2014. doi:10.1109/ICRA.2014.6907339. 9

    work page doi:10.1109/icra.2014.6907339 2014

  11. [11]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (ICML), vol- ume 139, pages 8748–8763. PMLR, 2021. URLhttps://proceedings.mlr.press/v139/ radford21a.html

    2021

  12. [12]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), volume 30, pages 6000–6010, 2017. URLhttps://proceedings.neurips. cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

    2017

  13. [13]

    Driess, F

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y . Chebotar, P. Sermanet, D. Duckworth, S. Levine, V . Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. InInternational Conference on Machine Learning ...

    2023

  14. [14]

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, H. Li, and T. Kong. Vision-language foundation models as effective robot imitators. InInternational Conference on Learning Representations (ICLR), 2024. URLhttps://openreview.net/ forum?id=lFYj0oibGR

    2024

  15. [15]

    Grannen, S

    J. Grannen, S. Karamcheti, S. Mirchandani, P. Liang, and D. Sadigh. V ocal sandbox: Contin- ual learning and adaptation for situated human-robot collaboration. InConference on Robot Learning (CoRL), volume 270 ofProceedings of Machine Learning Research. PMLR, 2024. URLhttps://proceedings.mlr.press/v270/grannen25a.html

    2024

  16. [16]

    Tziafas and H

    G. Tziafas and H. Kasaei. Lifelong robot library learning: Bootstrapping composable and generalizable skills for embodied control with language models. InIEEE International Con- ference on Robotics and Automation (ICRA), pages 515–522, 2024. doi:10.1109/ICRA57147. 2024.10611448

    work page doi:10.1109/icra57147 2024

  17. [17]

    W. Gu, S. Kondepudi, A. Gupta, L. Huang, and N. Gopalan. Continual robot skill and task learning via dialogue. InIEEE International Conference on Robotics and Automation (ICRA) Workshop on Human-Centered Robot Learning, 2025. URLhttps://openreview.net/ forum?id=r7PpkXMoVk

    2025

  18. [18]

    Paraschos, C

    A. Paraschos, C. Daniel, J. Peters, and G. Neumann. Probabilistic move- ment primitives. InAdvances in Neural Information Processing Systems (NeurIPS), 2013. URLhttps://proceedings.neurips.cc/paper/2013/hash/ e53a0a2978c28872a4505bdb51db06dc-Abstract.html

    2013

  19. [19]

    In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    J. Silv ´erio, Y . Huang, F. J. Abu-Dakka, L. Rozo, and D. G. Caldwell. Uncertainty-aware imi- tation learning using kernelized movement primitives. InIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 90–97, 2019. doi:10.1109/IROS40897.2019. 8967996

    work page doi:10.1109/iros40897.2019 2019

  20. [20]

    In: 2022 International Conference on Robotics and Automation (ICRA)

    P. Oikonomou, A. Dometios, M. Khamassi, and C. S. Tzafestas. Reproduction of human demonstrations with a soft-robotic arm based on a library of learned probabilistic movement primitives. In2022 International Conference on Robotics and Automation (ICRA), pages 5212–5218, 2022. doi:10.1109/ICRA46639.2022.9811627

    work page doi:10.1109/icra46639.2022.9811627 2022

  21. [21]

    Huang, J

    Y . Huang, J. Silv ´erio, L. Rozo, and D. G. Caldwell. Generalized task-parameterized skill learning. InIEEE International Conference on Robotics and Automation (ICRA), 2018. doi: 10.1109/ICRA.2018.8461079

    work page doi:10.1109/icra.2018.8461079 2018

  22. [22]

    J. Zhu, M. Gienger, and J. Kober. Learning task-parameterized skills from few demon- strations.IEEE Robotics and Automation Letters (RA-L), 7(2):4063–4070, 2022. doi: 10.1109/LRA.2022.3150013. 10

    work page doi:10.1109/lra.2022.3150013 2022

  23. [23]

    Hoyos, F

    J. Hoyos, F. Prieto, G. Aleny `a, and C. Torras. Incremental learning of skills in a task- parameterized gaussian mixture model.Journal of Intelligent & Robotic Systems, 82:81–99,

  24. [24]

    doi:10.1007/s10846-015-0290-3

    work page doi:10.1007/s10846-015-0290-3

  25. [25]

    Q. Team. Qwen3 technical report, 2025. doi:10.48550/arXiv.2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  26. [26]

    Z. Wang, Z. Cheng, H. Zhu, D. Fried, and G. Neubig. What are tools anyway? a survey from the language model perspective. InConference on Language Modeling (COLM), 2024. URL https://openreview.net/pdf?id=Xh1B90iBSR

    2024

  27. [27]

    Y . Qin, S. Hu, Y . Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, Y . Huang, C. Xiao, C. Han, et al. Tool learning with foundation models.ACM Computing Surveys (CSUR), 57:101:1–101:40,

  28. [28]

    T. Mai, R. Sakagami, G. Quere, G. Mesesan, R. Schuller, K. Fr ¨und, J. V ogel, A. Hagengruber, J. Lee, A. D ¨omel, F. Stulp, and S. Bustamante. LLM tool workflows for robot explainability and natural language commanding. InICRA 2026 Workshop on Semantics for Reliable Robot Autonomy: From Environment Understanding and Reasoning to Safe Interaction, 2026. U...

    2026

  29. [29]

    Ichter, A

    B. Ichter, A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y . Lu, C. Parada, K. Rao, P. Sermanet, A. T. To- shev, V . Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Lu...

    2023

  30. [30]

    N. Hogan. Impedance control of industrial robots.Robotics and Computer-Integrated Manu- facturing, 1(1):97–113, 1984. doi:10.1016/0736-5845(84)90084-X

    work page doi:10.1016/0736-5845(84)90084-x 1984

  31. [31]

    Burdick, and Aaron D

    M. Iskandar, C. Ott, O. Eiberger, M. Keppler, A. Albu-Sch ¨affer, and A. Dietrich. Joint-level control of the dlr lightweight robot sara. InIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8903–8910, 2020. doi:10.1109/IROS45743.2020.9340700

    work page doi:10.1109/iros45743.2020.9340700 2020

  32. [32]

    Iskandar, C

    M. Iskandar, C. Ott, A. Albu-Sch ¨affer, B. Siciliano, and A. Dietrich. Hybrid force-impedance control for fast end-effector motions.IEEE Robotics and Automation Letters (RA-L), 8(7): 3931–3938, 2023. doi:10.1109/LRA.2023.3270036

    work page doi:10.1109/lra.2023.3270036 2023

  33. [33]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. V...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.16054 2025

  34. [34]

    Strobl, Matthias Humt, and Rudolph Triebel

    M. Denninger, D. Winkelbauer, M. Sundermeyer, W. Boerdijk, M. Knauer, K. H. Strobl, M. Humt, and R. Triebel. Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software (JOSS), 8(82):4901, 2023. doi:10.21105/joss.04901

    work page doi:10.21105/joss.04901 2023

  35. [35]

    Vbench: Comprehensive benchmark suite for video generative models

    X. Long, Y .-C. Guo, C. Lin, Y . Liu, Z. Dou, L. Liu, Y . Ma, S.-H. Zhang, M. Habermann, C. Theobalt, and W. Wang. Wonder3D: Single Image to 3D Using Cross-Domain Diffusion. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9970–9980, 2024. doi:10.1109/CVPR52733.2024.00951

    work page doi:10.1109/cvpr52733.2024.00951 2024

  36. [36]

    Y . Yin, Z. Wang, Y . Sharma, D. Niu, T. Darrell, and R. Herzig. In-context learning enables robot action prediction in llms. InIEEE International Conference on Robotics and Automation (ICRA), 2025. doi:10.1109/ICRA55743.2025.11128807. 11

    work page doi:10.1109/icra55743.2025.11128807 2025

  37. [37]

    Certo, B

    A. Certo, B. Martins, C. Azevedo, and P. U. Lima. Large language model-based robot task planning from voice command transcriptions. InIEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), 2025. URLhttps://ieeexplore.ieee.org/document/ 11246378

    2025

  38. [38]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettle- moyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach them- selves to use tools. InAdvances in Neural Information Processing Systems (NeurIPS), vol- ume 36, 2023. URLhttps://proceedings.neurips.cc/paper_files/paper/2023/ hash/d842425e4bf79ba039352da0f65...

    2023

  39. [39]

    Huang and K

    J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey. InFind- ings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, 2023. doi:10.18653/v1/2023.findings-acl.67

    work page doi:10.18653/v1/2023.findings-acl.67 2023

  40. [40]

    W. Xu, M. Wang, W. Zhou, and H. Li. P-rag: Progressive retrieval augmented generation for planning on embodied everyday task. InACM International Conference on Multimedia (MM). ACM, 2024. doi:10.1145/3664647.3680661

    work page doi:10.1145/3664647.3680661 2024

  41. [41]

    arXiv preprint arXiv:2402.03610 , year=

    T. Kagaya, T. J. Yuan, Y . Lou, J. Karlekar, S. Pranata, A. Kinose, K. Oguri, F. Wick, and Y . You. Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents, 2024. doi:10.48550/arXiv.2402.03610

    work page doi:10.48550/arxiv.2402.03610 2024

  42. [42]

    Petruzzellis, C

    F. Petruzzellis, C. Cornelio, and P. Lio. Hierarchical planning for complex tasks with knowl- edge graph-rag and symbolic verification. InInternational Conference on Machine Learn- ing (ICML), volume 267 ofProceedings of Machine Learning Research. PMLR, 2025. URL https://proceedings.mlr.press/v267/petruzzellis25a.html

    2025

  43. [43]

    M. U. Din, J. Rosell, W. Akram, I. Zaplana, M. A. Roa, and I. Hussain. Llm-guided task and motion planning using knowledge-based reasoning, 2025. doi:10.48550/arXiv.2412.07493

    work page doi:10.48550/arxiv.2412.07493 2025

  44. [44]

    M. Lei, G. Wang, Y . Zhao, Z. Mai, Q. Zhao, Y . Guo, Z. Li, S. Cui, Y . Han, and J. Ren. Clea: Closed-loop embodied agent for enhancing task execution in dynamic environments, 2025. doi:10.48550/arXiv.2503.00729

    work page doi:10.48550/arxiv.2503.00729 2025

  45. [45]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V . Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning (CoRL), volume 229, pages 216...

    2023

  46. [46]

    A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation

    J. Barreiros, A. Beaulieu, A. Bhat, R. Cory, E. Cousineau, H. Dai, C.-H. Fang, K. Hashimoto, M. Z. Irshad, M. Itkina, N. Kuppuswamy, K.-H. Lee, K. Liu, D. McConachie, I. McMa- hon, H. Nishimura, C. Phillips-Grafflin, C. Richter, P. Shah, K. Srinivasan, B. Wulfe, C. Xu, M. Zhang, et al. A careful examination of large behavior models for multitask dexterous...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.05331 2025

  47. [47]

    M. J. Kim, C. Finn, and P. Liang. Fine-tuning vision-language-action models: Optimizing speed and success. InRobotics: Science and Systems (RSS), 2025. doi:10.15607/RSS.2025. XXI.017

    work page doi:10.15607/rss.2025 2025

  48. [48]

    C.-Y . Wang, A. Bochkovskiy, and H.-Y . M. Liao. Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7464–7475, 2023. doi:10.1109/CVPR52729.2023. 00721

    work page doi:10.1109/cvpr52729.2023 2023

  49. [49]

    Sundermeyer, Z.-C

    M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. InEuropean Conference on Computer Vision (ECCV), 2018. doi:10.1007/978-3-030-01231-1 43. 12

    work page doi:10.1007/978-3-030-01231-1 2018

  50. [50]

    Calli, A

    B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. Yale-cmu-berkeley dataset for robotic manipulation research.International Journal of Robotics Research (IJRR), 36(3):261–268, 2017. doi:10.1177/0278364917700714

    work page doi:10.1177/0278364917700714 2017

  51. [51]

    K. H. Strobl and G. Hirzinger. More accurate camera and hand-eye calibrations with un- known grid pattern dimensions. InIEEE International Conference on Robotics and Automation (ICRA), pages 1398–1405, 2008. doi:10.1109/ROBOT.2008.4543398

    work page doi:10.1109/robot.2008.4543398 2008

  52. [52]

    P. J. Besl and N. D. McKay. A method for registration of 3-D shapes.IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 14(2):239–256, 1992. doi:10.1109/34. 121791. 13 A Supplementary Material This appendix provides supplementary material. Section A.1 provides the extended gap-in-literature discussion. Section A.2 provides additional r...

    work page doi:10.1109/34 1992

  53. [53]

    skill name

    acquires and composes learned visuo-motor policies through dialogue-based interaction. These approaches compose skillssymbolically, selecting and sequencing discrete primitives rather than operating at the continuous trajectory level. Trajectory-level fusion via products of Gaussians is an established mechanism in probabilistic and kernelized movement pri...