Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

ROSA: Harnessing Robot States for Vision-Language and Action Alignment

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Auxiliary robot state estimation, trained with random-motion data, narrows the vision-language-to-action alignment gap and delivers its largest gains when expert demonstrations are scarce.

desk verdict Useful low-data VLA recipe undermined by a missing control: random data is never tested as action data, so the state label's causal role is unproven. read the letter →

arxiv 2506.13679 v1 pith:I66ZQJNT submitted 2025-06-16 cs.RO cs.AIcs.CV

Yuqing Wen , Kefan Gu , Haoxuan Liu , Yucheng Zhao , Tiancai Wang , Haoqiang Fan , Xiaoyan Sun This is my paper
classification cs.ROcs.AIcs.CV
keywords vision-language-actionmodelsrobotstateestimationauxiliarysupervisiondataefficiencyspatio-temporalalignmentroboticmanipulationRLBenchgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the alignment problem in vision-language-action models can be attacked with an auxiliary task that costs almost nothing. Instead of only fine-tuning a VLM on expert demonstrations of what the robot should do next, ROSA also asks the model to estimate the robot's current 3D end-effector pose and gripper state from the same image. Because the state has the same format as the action, one next-token-prediction objective supervises both, and the state data can be collected automatically by letting the robot move randomly. The paper reports large gains in low-data regimes (7.1 and 11.4 percentage points on RLBench with 50 and 100 demos) and a near-doubling of real-robot success rates, suggesting that cheap self-state supervision is a practical route to data-efficient VLA training.

What carries the argument

The key mechanism is an auxiliary robot state estimation task built on format-identical data. ROSA records pairs of an image, the uniform language instruction 'What is the current state of the robot?', and a 7-degree-of-freedom state (position, Euler angles, and gripper open/close) by commanding random robot motions within a hand-defined feasible action space, then mixes them with expert action pairs at a 1:4 ratio. Because actions and states share the same 7-dimensional format, the same linear quantization and autoregressive cross-entropy loss handles both, so the model learns to emit a current state and a next action from the same visual input. Ablations show that the 1:4 mixing ratio and a moderate number of scenes matter, while the relevance of the scene content does not.

What would settle it

Collect state data from a feasible space deliberately chosen to avoid the regions of the action space that expert demonstrations occupy (for example, only the far corners of the workspace) and train the same model with the same 1:4 ratio; if the success rate stays at or above the expert-only baseline, the benefit is not due to distributional overlay, and if it collapses, the assumption of distributional compatibility is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a VLA model trained jointly on expert action prediction and robot state estimation outperforms the same model trained on expert data alone, with the largest margin when expert data is scarce. The paper attributes this to closing a spatio-temporal gap: VLM features are semantic and present-oriented, while actions are 3D and future-oriented; state estimation forces the model to produce fine-grained 3D self-awareness, which then supports action prediction. The evidence includes a linear-probe test where ROSA reaches 92% accuracy on 3D understanding versus 61% for the baseline, and real-world generalization trials where average success on unseen tasks rises from 43% to 85%.

Load-bearing premise

The claim rests on the assumption that random state data collected inside a hand-defined feasible action space, mixed with expert data at a 1:4 ratio, is similar enough to expert actions that the auxiliary task improves action prediction instead of corrupting it.

Editorial extensions

If this is right

  • With 50 or 100 expert demonstrations per task on RLBench, ROSA improves average success rate by 7.1 and 11.4 percentage points over the same baseline, showing that the auxiliary task is most valuable exactly where data is scarce.
  • On the real robot, ROSA roughly doubles the average success rate in low-data settings and raises average generalization success on unseen tasks from 43% to 85%, suggesting improved robustness to novel objects, containers, and distractors.
  • In one-shot training, ROSA achieves non-zero success rates on three RLBench tasks where the baseline scores zero on all of them, indicating that state estimation can bootstrap action prediction from a single expert sample.
  • The benefit persists even at 500 demonstrations per task (a 1.6-point gain), so the auxiliary state task does not merely substitute for data but continues to help when data is abundant, though the marginal gain shrinks.
  • Adding too much state data degrades performance: at a 1:2 state-to-action ratio the success rate falls from 63.7% to 58.7%, which implies the alignment benefit comes from a careful balance rather than simply more state supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is mostly distributional coverage rather than self-awareness, then the random state data could be replaced by any cheap off-policy or synthetically sampled action-covering data; a direct test would compare same-format pseudo-actions sampled from a fitted action prior against the recorded random states.
  • The paper shows scene relevance does not matter for state collection, so the state data could be collected in empty or unrelated environments; this opens a path to cross-embodiment or cross-camera transfer if the fixed camera-robot pose assumption is relaxed.
  • Because the gains shrink as expert data grows, a natural extension is a data-mixing schedule that anneals the state-data ratio down as expert demonstrations accumulate, which the paper does not explore.
  • The single fixed viewpoint limitation noted in the appendix suggests a testable extension: collecting state data under varying camera positions could create a self-supervised spatial-consistency objective, potentially improving spatial reasoning further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes ROSA, a training paradigm for vision-language-action (VLA) models that adds robot state estimation data to standard expert action data. The state estimation data is collected automatically by letting a robot perform random movements within a constrained scene, recording observations paired with the current end-effector pose and gripper status under a fixed language instruction. This data is mixed with expert demonstrations in a 1:4 ratio and used to fine-tune a Qwen-2.5-7B based VLA model with a unified next-token-prediction loss. The authors report consistent improvements over a baseline on 12 RLBench tasks at five data scales, with the largest gains at 50 and 100 demonstrations per task (7.1 and 11.4 percentage points, respectively), plus real-robot results on seen and unseen tasks, one-shot results, ablations on the state-data ratio and scene relevance, and a linear-probing analysis of 3D understanding.

Significance. If the central claim is established, the paper would be a practical contribution to data-efficient VLA training: it shows that cheap, label-free state supervision can improve action prediction at low data regimes, with controlled comparisons at multiple scales, real-robot generalization experiments, and a credible architectural story motivated by the spatial and temporal gaps in VLM-to-VLA adaptation. The automated data collection pipeline and the explicit comparison with prior methods on RLBench are useful for the community. However, the causal attribution of the gains to the state-estimation objective rather than to the extra random data is not yet demonstrated, and this is the main weakness of the paper.

major comments (3)
  1. [Sec. 3.1 and Tab. 1] The reported gains conflate two interventions: the robot state data differs from expert action data in trajectory distribution (random vs. expert), target-label semantics (current state vs. next action), and language instruction. Because the random trajectories are recorded as time series, the same observations can be relabeled as action-prediction data by shifting the end-effector pose one step forward and using the expert instruction. Without this control, the improvements in Tab. 1 could be caused by adding diverse random action data rather than by the current-state estimation objective, which is the paper's stated mechanism. Please add this relabeled-data control, or an equivalent ablation that isolates the label semantics while holding the data distribution fixed, for at least the 50- and 100-demonstration settings.
  2. [Sec. 4.2.4 and Tab. 7] The linear-probe analysis cannot discriminate between the two factors in the training-mixture confound. ROSA is trained on the state-estimation objective, and the probe is trained on the same objective, so it is unsurprising that ROSA's features decode pose better; the table does not show whether the benefit comes from the auxiliary objective or from the extra random coverage of the action space. Interpret this result as a mechanism illustration rather than causal evidence, or couple it with the relabeled-data control.
  3. [Sec. 4.1 and Fig. 5] Real-robot results are reported without error bars or trial-level variance, and the one-shot results in Tab. 2 are reported without variance or evaluation details. Given that the paper's headline real-world claim of a 35% average success-rate improvement rests on these curves, please report the number of evaluation runs, per-seed or per-trial breakdowns, and standard deviations for the real-robot experiments and the one-shot experiments.
minor comments (7)
  1. [Sec. 1, Contributions] There is a typo in the second contribution: 'creat' should be 'create'.
  2. [Sec. 4.2.4, Tab. 7] The table header reads 'Liner-prob evaluation' and should be 'Linear-prob evaluation'.
  3. [Eq. (5)] The sentence following Eq. (5) says 'ω demotes the parameters' but should read 'denotes'.
  4. [Sec. 3.3 and Sec. A.3] 'epoches' should be 'epochs' in both places.
  5. [Sec. 4.2.4 and Sec. 4.3] Tab. 6 refers to 'relevant' and 'irrelevant' scenes, but these terms are only defined later in Sec. 4.3; please define them at first use.
  6. [Sec. 3.1 and Fig. 2] The 'feasible action space' for random state-data collection is described only qualitatively; please specify how the bounds are chosen and how collisions with scene objects are avoided, since this affects the distribution of the collected state data.
  7. [Sec. 4.2.2 and Tab. 3] The four unseen real-robot tasks are evaluated over 10 trials, but the paper does not state whether the same trials are used across methods or how object placements vary; please clarify the evaluation protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No evidence of circularity; the central claim is anchored by held-out task success on RLBench and a real robot.

full rationale

The paper's central claim is empirical: ROSA changes the training-data mixture by adding automatically collected robot-state samples and then measures action success on held-out RLBench tasks and real-robot trials. These task-success metrics are external to the training objective and are not derived from the paper's own equations, so the main result does not reduce to its inputs. The design is not definitionally circular: robot states and expert actions share the same 7-DoF format by intentional construction, but state estimation is an auxiliary supervision signal, not a derivation of action prediction. The linear-probe analysis in Sec. 4.2.4 is partly self-confirming because ROSA is explicitly trained to predict the same state quantities that the probe evaluates, but this probe is used only as a mechanism check; the action-prediction gains in Tabs. 1, 3, and 4 are independently measured held-out success rates. The paper's limitation section notes scope restrictions such as fixed camera viewpoint and fixed relative pose, but it does not conceal a circular derivation. The missing control separating extra random data from state-estimation supervision is a validity concern about causal attribution, not a circularity of the kind where a predicted quantity is equivalent by construction to a fitted input.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper's contribution is empirical, so the main free parameters are training and collection choices tuned via ablations (state data ratio, scene count) or inherited from prior work (keyframe thresholds, token quantization). The important background assumptions are domain assumptions about demonstration quality, visual observability of robot state, and joint trainability of the two tasks.

free parameters (4)
  • state-to-action data mixing ratio = 1:4
    Ablation in Tab 5: 1/4 ratio gives best RLBench success among 0, 1/8, 1/4, 1/2; larger amounts degrade performance.
  • number of state-collection scenes = 100
    Ablation in Tab 6: 100 scenes yields 63.7% SR, highest among 50/100/200; 200 scenes slightly degrade performance.
  • action token bin_size = 256
    Chosen by hand in Sec 3.2 for linear quantization of continuous poses and gripper state; no sensitivity analysis is provided.
  • keyframe extraction thresholds (epsilon, recent_buffer size) = epsilon and buffer size 4
    Algorithm 1 in Sec A.3; thresholds inherited from PerAct [14] and not tuned for this paper, yet they determine which action samples are used for training.
assumptions (3)
  • domain assumption Expert demonstrations provide optimal or near-optimal action supervision for the target tasks.
    Standard imitation learning assumption; used throughout Sec 3.1 and Sec 4.1 to justify using expert data as the action prediction target.
  • domain assumption The robot's recorded internal state (end-effector pose, gripper open/close) is the correct supervision signal for the state estimation task and is recoverable from a single front RGB image.
    Sec A.2 logs pose relative to robot base; the model is trained to predict it from the image, which requires that the image is informative enough and that the logged state is treated as ground truth.
  • domain assumption A pretrained 7B LLM (Qwen-2.5-7B) with CLIP ViT-L/14 can learn both action prediction and state prediction in a shared output space without harmful interference.
    Sec 3.3 uses full fine-tuning for 6-9 epochs; no evidence is offered for why the two tasks do not interfere, and the ratio ablation suggests interference when state data is too abundant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ROSA: Harnessing Robot States for Vision-Language and Action Alignment." pith.science (2026). https://pith.science/paper/I66ZQJNT

@misc{pith2026250613679,
  author       = {Pith},
  title        = {Pith review of: ROSA: Harnessing Robot States for Vision-Language and Action Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I66ZQJNT}},
  note         = {Machine review of arXiv:2506.13679}
}
read the original abstract

Vision-Language-Action (VLA) models have recently made significant advance in multi-task, end-to-end robotic control, due to the strong generalization capabilities of Vision-Language Models (VLMs). A fundamental challenge in developing such models is effectively aligning the vision-language space with the robotic action space. Existing approaches typically rely on directly fine-tuning VLMs using expert demonstrations. However, this strategy suffers from a spatio-temporal gap, resulting in considerable data inefficiency and heavy reliance on human labor. Spatially, VLMs operate within a high-level semantic space, whereas robotic actions are grounded in low-level 3D physical space; temporally, VLMs primarily interpret the present, while VLA models anticipate future actions. To overcome these challenges, we propose a novel training paradigm, ROSA, which leverages robot state estimation to improve alignment between vision-language and action spaces. By integrating robot state estimation data obtained via an automated process, ROSA enables the VLA model to gain enhanced spatial understanding and self-awareness, thereby boosting performance and generalization. Extensive experiments in both simulated and real-world environments demonstrate the effectiveness of ROSA, particularly in low-data regimes.

Figures

Figures reproduced from arXiv: 2506.13679 by the authors.

Figure 1
Figure 1. The spatial and temporal gaps in adapting VLMs to VLAs. VLMs are pretrained with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the two types of data used by ROSA to train VLA models. (a). Expert action [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the ROSA architecture. ROSA adopts a classic VLM architecture. Image [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Task examples for RLBench and real-world robot. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison of performance between ROSA and the baseline under varying data scales on [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visual examples of ROSA on RLBench and real-world robot tasks. The white number in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Examples of robot state data. Visualizations of Robot State Data: The robot state data is collected by allowing the robotic arm to perform random movements within its valid action space. We set up different scenarios for collecting such data and the examples are il￾lus…
Figure 8
Figure 8. Figure 8: Real-robot setup with WidowX 250s and D435i. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Visual examples of ROSA on seen tasks on real-world robot. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Visual examples of ROSA on unseen tasks on real-world robot. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Visual examples of ROSA on long-horizon unseen tasks on real-world robot. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Visual examples of ROSA on interference tasks on real-world robot. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models

    cs.RO 2026-08 conditional novelty 7.0 of 10

    Robots keep the right task in mind under paraphrased instructions but execute wrong actions because joint vision-language features shift; rerouting semantics through frozen T5 and retraining the action expert restores...

Reference graph

Works this paper leans on

46 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  2. [2]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  3. [3]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    arXiv 2024

  4. [4]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. InForty-first International Conference on Machine Learning, 2024

    2024

  5. [5]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  6. [6]

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

    arXiv 2024

  7. [7]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    arXiv 2024

  8. [8]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. InTowards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023, 2023

    work page 2023

Show all 46 references
  1. [9]

    pi_0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi_0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    arXiv

  2. [10]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  3. [11]

    Chatvla: Unified multimodal understanding and robot control with vision-language-action model.arXiv preprint arXiv:2502.14420, 2025

    Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-action model.arXiv preprint arXiv:2502.14420, 2025

    2025 arXiv

  4. [12]

    Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823, 2024

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823, 2024

    2024 arXiv

  5. [13]

    Octo: An open-source generalist robot policy

    Oier Mees, Dibya Ghosh, Karl Pertsch, Kevin Black, Homer Rich Walke, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, et al. Octo: An open-source generalist robot policy. InFirst Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024...

    2024

  6. [14]

    Perceiver-actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. InConference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  7. [15]

    Vision-language foundation models as effective robot imitators.arXiv preprint arXiv:2311.01378, 2023

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators.arXiv preprint arXiv:2311.01378, 2023

    2023 arXiv

  8. [16]

    Rvt: Robotic view transformer for 3d object manipulation

    Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. InConference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  9. [17]

    Rvt-2: Learning precise manipulation from few demonstrations.arXiv preprint arXiv:2406.08545, 2024

    Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, and Dieter Fox. Rvt-2: Learning precise manipulation from few demonstrations.arXiv preprint arXiv:2406.08545, 2024

    2024 arXiv

  10. [18]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. InConference on Robot Learning, pages 991–1002. PMLR, 2022

    2022

  11. [19]

    Coarse-to-fine q- attention: Efficient learning for visual robotic manipulation via discretisation

    Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J Davison. Coarse-to-fine q- attention: Efficient learning for visual robotic manipulation via discretisation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739–13748, 2022

    2022

  12. [20]

    Robot learning with sensorimotor pre-training

    Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. InConference on Robot Learning, pages 683–693. PMLR, 2023

    2023

  13. [21]

    Vima: General robot manipulation with multimodal prompts.arXiv preprint arXiv:2210.03094, 2(3):6, 2022

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts.arXiv preprint arXiv:2210.03094, 2(3):6, 2022

    arXiv

  14. [22]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  15. [23]

    Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    2023 arXiv

  16. [24]

    Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  18. [26]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  19. [27]

    Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024

    2024 arXiv

  20. [28]

    Towards generalist robot policies: What matters in building vision-language-action models.arXiv preprint arXiv:2412.14058, 2024

    Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, Hanbo Zhang, and Huaping Liu. Towards generalist robot policies: What matters in building vision-language-action models.arXiv preprint arXiv:2412.14058, 2024

    2024 arXiv

  21. [29]

    Llarva: Vision-action instruction tuning enhances robot learning

    Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. In8th Annual Conference on Robot Learning. 11

  22. [30]

    Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies.arXiv preprint arXiv:2412.10345, 2024

    Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies.arXiv preprint arXiv:2412.10345, 2024

    2024 arXiv

  23. [31]

    Llara: Supercharging robot learning data for vision-language policy.arXiv preprint arXiv:2406.20095, 2024

    Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jae Lee, et al. Llara: Supercharging robot learning data for vision-language policy.arXiv preprint arXiv:2406.20095, 2024

    2024 arXiv

  24. [32]

    Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation.IEEE Robotics and Automation Letters, 2025

    Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation.IEEE Robotics and Automation Letters, 2025

    2025

  25. [33]

    Towards fast, memory-based and data-efficient vision-language policy.arXiv preprint arXiv:2503.10322, 2025

    Haoxuan Li, Sixu Yan, Yuhan Li, and Xinggang Wang. Towards fast, memory-based and data-efficient vision-language policy.arXiv preprint arXiv:2503.10322, 2025

    2025 arXiv

  26. [34]

    Mole-vla: Dynamic layer-skipping vision language action model via mixture-of-layers for efficient robot manipulation.arXiv preprint arXiv:2503.20384, 2025

    Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, and Shanghang Zhang. Mole-vla: Dynamic layer-skipping vision language action model via mixture-of-layers for efficient robot manipulation.arXiv preprint arXiv:2503.20384, 2025

    2025 arXiv

  27. [35]

    An atomic skill library construction method for data-efficient embodied manipulation.arXiv preprint arXiv:2501.15068, 2025

    Dongjiang Li, Bo Peng, Chang Li, Ning Qiao, Qi Zheng, Lei Sun, Yusen Qin, Bangguo Li, Yifeng Luan, Bo Wu, et al. An atomic skill library construction method for data-efficient embodied manipulation.arXiv preprint arXiv:2501.15068, 2025

    2025 arXiv

  28. [36]

    Pose estimation for an autonomous vehicle using monocular vision

    Nikunj Kothari, Misha Gupta, Leena Vachhani, and Hemendra Arya. Pose estimation for an autonomous vehicle using monocular vision. In2017 Indian control conference (ICC), pages 424–431. IEEE, 2017

    2017

  29. [37]

    Pose estimation and map building with a time-of-flight-camera for robot navigation.International Journal of Intelligent Systems Technologies and Applications, 5(3-4):355–364, 2008

    A Prusak, O Melnychuk, H Roth, Ingo Schiller, and Reinhard Koch. Pose estimation and map building with a time-of-flight-camera for robot navigation.International Journal of Intelligent Systems Technologies and Applications, 5(3-4):355–364, 2008

    2008

  30. [38]

    Learning human-to-robot handovers from point clouds

    Sammy Christen, Wei Yang, Claudia Pérez-D’Arpino, Otmar Hilliges, Dieter Fox, and Yu-Wei Chao. Learning human-to-robot handovers from point clouds. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9654–9664, 2023

    2023

  31. [39]

    Pose estima- tion and adaptive robot behaviour for human-robot interaction

    Mikael Svenstrup, Soren Tranberg, Hans Jorgen Andersen, and Thomas Bak. Pose estima- tion and adaptive robot behaviour for human-robot interaction. In2009 IEEE International Conference on Robotics and Automation, pages 3571–3576. IEEE, 2009

    2009

  32. [40]

    Privacy-preserving pose estimation for human-robot interaction.arXiv preprint arXiv:2011.07387, 2020

    Youya Xia, Yifan Tang, Yuhan Hu, and Guy Hoffman. Privacy-preserving pose estimation for human-robot interaction.arXiv preprint arXiv:2011.07387, 2020

    2011 arXiv

  33. [41]

    3d robot pose estimation from 2d images.arXiv preprint arXiv:1902.04987, 2019

    Christoph Heindl, Sebastian Zambal, Thomas Ponitz, Andreas Pichler, and Josef Scharinger. 3d robot pose estimation from 2d images.arXiv preprint arXiv:1902.04987, 2019

    1902 arXiv

  34. [42]

    Multi-objective convolutional neural networks for robot localisation and 3d position estimation in 2d camera images

    Justinas Miseikis, Inka Brijacak, Saeed Yahyanejad, Kyrre Glette, Ole Jakob Elle, and Jim Torresen. Multi-objective convolutional neural networks for robot localisation and 3d position estimation in 2d camera images. In2018 15th International Conference on Ubiquitous Robots (U...

    2018

  35. [43]

    Robots’ state estimation and observability analysis based on statistical motion models.IEEE Transactions on Control Systems Technology, 30(5):2030–2045, 2022

    Wei Xu, Dongjiao He, Yixi Cai, and Fu Zhang. Robots’ state estimation and observability analysis based on statistical motion models.IEEE Transactions on Control Systems Technology, 30(5):2030–2045, 2022

    2022

  36. [44]

    Real-time holistic robot pose estimation with unknown states

    Shikun Ban, Juling Fan, Xiaoxuan Ma, Wentao Zhu, Yu Qiao, and Yizhou Wang. Real-time holistic robot pose estimation with unknown states. InEuropean Conference on Computer Vision, pages 1–17. Springer, 2024

    2024

  37. [45]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  38. [46]

    push the maroon button, then push the green button

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment.IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020. 12 Appendix In the appendix, we provide more detailed descriptions of the experimental ...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.