REVIEW 4 major objections 5 minor 80 references
JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that weakly labeled human egocentric video, when aligned through implicit latent actions and an explicit canonical action space, can serve as a primary scaling axis for real-world robot manipulation.
desk verdict Dual action alignment is a coherent architectural contribution with a plausible scaling story, but the headline scaling claims rest on three points, no variance, and an unspecified supervision mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Latent-Action-Conditioned World Model (LAC-WM), a world model pretrained with a variational latent-action model that infers a latent action from consecutive multi-view observations and uses those latent actions as transition-level conditioning, decomposing future prediction into transition-specific modes. Complementing it is the explicit alignment channel: a 130-dimensional canonical state-action representation with camera-frame chunk-relative end-effector actions, which maps human, simulation, and robot trajectories into a unified physical action space used by a flow-matching action expert. An inner-outer-loop reinforcement-learning stage then pairs rapid residual adaptation with periodic foundation-policy improvement.
What would settle it
A discriminating experiment: rerun the EgoLive scaling study at 10%, 25%, 50%, and 100% with the action expert trained with no action or latent supervision at all, using visual next-frame prediction only; if the same monotonic rise from 47.8 to 85.6 appears, the scaling effect does not depend on action alignment. A second check: repeat the benchmark many times per task to obtain confidence intervals; if the four-point trend disappears within noise, the 'no plateau' conclusion is unsupported.
Extended reading notes
Core claim
The central discovery is that naively pooling heterogeneous data causes negative transfer, but when each data source is routed to the supervision it can reliably support, the same data become complementary. Action-free human, simulation, and robot video teaches physical dynamics through a shared latent-action space inferred from visual transitions, while trajectories with reliable motion annotations are mapped into a unified physical action space for executable control. The paper reports consistent scaling on real hardware: when only the human-video fraction changes from 10% to 100%, seen-task scores rise from 83.1 to 97.5 and unseen-task scores from 56.9 to 72.4, while robot data and downstream training remain fixed. This is taken as evidence that human egocentric video is not merely auxiliary data but a primary axis along which manipulation capability can be scaled.
Load-bearing premise
The load-bearing premise is that the observed scaling really comes from the dual-alignment training signal on human video; the paper does not say what supervision the action expert receives in its pure-ego pretraining study, and the benchmark is self-created with no error bars, so the mechanism and the precision of the trend are unverified.
Editorial extensions
If this is right
- If the scaling result is right, human egocentric video can substitute for a large fraction of expensive robot demonstration data in pretraining.
- Continuing to grow the human-video corpus beyond 53k hours should further raise seen and unseen task scores, since the last increment still improved performance.
- Implicit alignment should keep being the main contributor to robustness under background and illumination shifts, while explicit alignment should keep being essential for precise, long-horizon execution.
- The frozen world model's dynamics features, learned without action labels, transfer to downstream executable action prediction.
- Cross-embodiment training through the unified 130-dimensional action space avoids the negative transfer that naive pooling produces.
Reading between the lines
- If the mechanism is what drives the gain, pretraining on unrelated egocentric video, such as cooking and assembly tutorials, should produce similar task-score gains on the same robot; this is directly testable and not reported in the paper.
- The monotonic trend is reported as four points without confidence intervals; re-running the scaling study with repeated trials and error bars would show how much of the 47.8-to-85.6 jump is signal versus noise.
- The pure-ego study does not state what supervision the action expert receives from ego video, leaving open the possibility that visual-semantic adaptation alone, rather than explicit alignment, causes the scaling; varying the supervision type is a discriminating experiment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes JoyAI-RA 0.5, a Vision-Language-World-Action (VLWA) framework for robot manipulation that combines a VLM, a latent-action-conditioned world model (LAC-WM), and a flow-matching action expert. The central idea is a dual action alignment: implicit alignment infers latent actions from visual transitions to leverage action-free human/simulation/robot video for world-model pretraining, and explicit alignment maps reliable human and robot trajectories into a 130-dimensional canonical action space with camera-frame chunk-relative end-effector actions. Training proceeds in four stages: LAC-WM pretraining, cross-embodiment VLWA pretraining, target-robot post-training, and inner-outer-loop RL. The paper evaluates on a self-created Real-World AgiBot Benchmark on the AgiBot G1 platform, reporting strong seen-task performance, competitive unseen generalization, and a scaling trend with human egocentric video. The headline claim is that human egocentric video is not merely auxiliary but a primary axis for scaling real-world manipulation.
Significance. If the scaling claim is substantiated, the paper would make an important empirical contribution: it would demonstrate that large-scale, weakly labeled human egocentric video can be converted into transferable learning signal for real robot control, and that performance continues to improve with more such data. The work is also strong in scope and effort: it integrates over 70K hours of heterogeneous data, conducts real-robot evaluations across multiple task families, includes systematic ablations that separately remove implicit and explicit alignment, and reports a component-level scaling study. The evaluation is task-completion based and therefore not circular with the training objective. However, the central claim currently rests on very few data points without uncertainty quantification, and a key pretraining configuration (Sec 5.4.2) leaves the action supervision for ego video unspecified. These issues must be addressed before the headline conclusion can be considered established.
major comments (4)
- [Sec. 5.4.3, Fig. 11] The paper's central claim that human egocentric video is a primary scaling axis is supported by only three point estimates (10%, 25%, 100% human-video fraction), with no error bars, confidence intervals, number of seeds, or per-task breakdown. The reported task score is described in Sec. 5.1 as a mean over 20 seen and 10 unseen trials, but the formula for aggregating subtask completion rates is not given, and no measure of dispersion is reported anywhere in the paper, including Figs. 7, 8, 10, and Tables 2 and 3. With 10 unseen trials, differences such as the 56.9 to 67.7 jump could be within trial-to-trial noise. The 50% condition is also omitted from Fig. 11 even though it appears in Fig. 9, so the claimed monotonicity is an assertion about three points. The benchmark is self-created and not released, making independent verification impossible. I request that the authors report per-task scores with confidence intervals, run multiple seeds or at least provide variance across trials, include the 50% point or explain its omission, and release the benchmark evaluation protocol and trajectories.
- [Sec. 5.4.2, Fig. 10] The EgoLive scaling study does not specify what supervision the action expert receives when it is 'fully trained on the EgoLive dataset' while the world model remains frozen. Section 4.2.2 explicitly states that egocentric clips without reliable physical trajectories do not receive explicit action supervision, and Sec. 3.2 says clips with unreliable or missing hand poses are retained only for latent-action-based LAC-WM pretraining. The paper never states the fraction of EgoLive clips that pass the reliability threshold, nor whether the action expert in this study is trained with recovered hand trajectories, canonical actions, latent actions, or no action supervision at all. Without this information, the seen-task improvement from 47.8 to 85.6 in Fig. 10 cannot be attributed to action alignment; it may reflect other cues in the ego video (e.g., visual diversity or VLM semantic learning). Please specify the exact loss terms and supervision fractions used in this experiment.
- [Sec. 5.2.1, Fig. 7] The main comparison is against a single baseline, π0.5, with no details on which checkpoint or variant was used, how it was adapted to the AgiBot G1 platform and the 130-dimensional action space, how many trials were run, or what variance was observed. The text says 'under an identical evaluation protocol' but the protocol itself is not described beyond trial counts. Given that the baseline is used to support claims such as 'JoyAI-RA 0.5 clearly outperforms π0.5' and the only exception in spatial-topological generalization, at least per-cell trial counts and confidence intervals are needed. If the baseline was used as provided with its original action space, the comparison may not be meaningful; please document the adaptation procedure.
- [Sec. 5.1, Tables 2 and 3] Several ablation conclusions are based on small absolute differences: for example, Table 3 reports a 4.8-point gain from latent-action conditioning (87.3 vs 92.1), and Table 2 reports a 3.0-point seen and 3.1-point unseen gain from the world model. With only a single mean over 20 and 10 trials, these differences could be within noise. Since these ablations underpin the claim that both alignment mechanisms and the world model contribute, I ask for per-trial data, confidence intervals, and ideally a small number of independent evaluation runs (or at least a bootstrap over trials).
minor comments (5)
- [Sec. 5.3.2, Table 2] The 'w/ WM' row reads '62.3 40.651.5' with the average 51.5 concatenated to the unseen score; there is a missing space or delimiter.
- [Sec. 5.4.3, Fig. 11] The figure legend and text describe fractions as 'approximately 10%' and 'approximately 25%' while the axis labels read exactly 10%, 25%, and 100%. Please clarify whether these are exact nested subsets or approximate fractions, and state how the subsets are sampled.
- [Sec. 3.3.2, Eq. (5)] The notation in Eq. (5) defines W TCt as the transform from the conditioning camera frame to the reference frame, but the text says 'using a fixed trajectory-level reference frame W to relate camera poses across timestamps.' Please define the frame W explicitly and state whether it is a world frame, the first camera frame, or something else.
- [Sec. 5.2.2, Fig. 8] The figure reports task success rates for mouse and headphone tasks, but the y-axis and text do not indicate the number of trials per condition; please add trial counts and, where possible, confidence intervals.
- [Sec. 5.4.1, Fig. 9] The t-SNE visualization includes a 50% scale, but the subsequent policy-scaling experiment in Fig. 10 includes 50% while Fig. 11 does not; please explain this discrepancy in the text.
Circularity Check
No derivation-reducing circularity: the scaling claim is measured by external real-robot task scores, with only minor non-load-bearing self-citations to EgoLive [38] and JoyAI-RA 0.1 [75].
full rationale
JoyAI-RA 0.5's central derivation chain is not circular. The scaling claims in Sec. 5.4.2 and Sec. 5.4.3 are evaluated with downstream real-robot task scores after an identical robot post-training protocol, not with the pretraining objectives of Eq. (3), Eq. (11), or Eq. (14); therefore no fitted parameter or pretraining loss is renamed as a prediction. The LAC-WM scaling study (Sec. 5.4.3) holds the robot trajectory corpus fixed and varies only the human-video fraction, and the EgoLive scaling study (Sec. 5.4.2) freezes the world model while training the VLM and action expert on nested ego-video subsets, with the final metric being physical task execution on the AgiBot G1. The dual-alignment ablations (Sec. 5.3.1) compare plausible controls, such as an off-the-shelf world model without latent-action conditioning and direct action-dimension concatenation without the canonical action space, under an identical downstream protocol, so the reported gains are not forced by construction. The paper does cite its own EgoLive dataset [38] for the scaling corpus and its own JoyAI-RA 0.1 [75] for the flow-matching action-expert architecture, but neither citation imports an unverified uniqueness theorem or defines the benchmark in terms of the cited work's outputs; the downstream task score is external to both. The under-specified supervision for the action expert in the EgoLive scaling study, the missing error bars, and the unreleased self-created benchmark are statistical and reproducibility concerns, not circularity. Score 2 reflects only the two minor self-citations, which are not load-bearing for the central claim.
Assumptions & free parameters
free parameters (4)
- Reconstruction loss weights λr and KL weight β_KL (Eq. 3) =
not reported
- Stage 2 loss weights λ_VQA, λ_FAST, λ_FM (Eq. 15) =
not reported
- Condition-dropout probability p_drop =
not reported
- Action chunk horizon H and world-model horizon H_v =
not reported
assumptions (5)
- domain assumption Latent actions inferred from visual transitions are embodiment-agnostic and transferable across human, simulation, and robot data.
- domain assumption The quality metrics (derivative statistics, LDLJ, PSD, spectral arc length, motion spikes, temporal entropy) identify demonstrations whose removal improves downstream learning.
- domain assumption The camera-frame chunk-relative end-effector representation preserves manipulation motion across embodiments.
- ad hoc to paper The 130-dimensional canonical action space with fixed slot ordering provides sufficient physical coverage for cross-embodiment transfer.
- standard math Flow matching with a linear interpolation path (Eq. 10) provides a valid generative objective for both future-video latents and action chunks.
Cite this review
Pith. "Pith review of JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment." pith.science (2026). https://pith.science/paper/SHKICYWQ
@misc{pith2026260805674,
author = {Pith},
title = {Pith review of: JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/SHKICYWQ}},
note = {Machine review of arXiv:2608.05674}
}
read the original abstract
Robot data is scarce, so generalist policies need to learn from heterogeneous sources, including human egocentric video, simulation, and real robots, which differ in supervision and embodiment, with action labels missing or mutually incompatible. Human egocentric data scale best but sit farthest from robot data, and naive pooling causes negative transfer rather than knowledge sharing. We propose JoyAI-RA 0.5, a generalist Vision-Language-World-Action (VLWA) framework that couples physical world-dynamics priors with visual semantics and scales manipulation learning across such data via dual action alignment. Implicit action alignment infers latent actions from visual transitions, enabling action-free human, simulation, and robot data to guide a latent-action-conditioned world model in learning physical dynamics. Explicit alignment grounds reliable human and robot trajectories in a unified physical action space through a canonical action representation and camera-frame chunk-relative end-effector actions. An inner-outer-loop reinforcement stage then pairs efficient task adaptation with foundation-policy improvement. On a real-world AgiBot benchmark, JoyAI-RA performs strongly on both seen tasks and unseen variations. The task score improves consistently as the volume of human egocentric pretraining data increases and shows no sign of plateauing at our largest scale. This suggests that abundant but weakly labeled human experience can be converted into a transferable training signal, making human video not merely a weak auxiliary source but a primary axis along which manipulation capability can be scaled. Project page can be found at https://joyai-ra-05.github.io/.
Reference graph
Works this paper leans on
-
[1]
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023
arXiv 2023
-
[2]
AgiBot-World Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025. URLhttps://arxiv.org/abs/2503.06669
arXiv 2025
-
[3]
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025
arXiv 2025
-
[4]
Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030, 2025
Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, et al. Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030, 2025
arXiv 2025
-
[5]
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025
arXiv 2025
-
[6]
arXiv preprint arXiv:2410.24164, 2024
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024
-
[7]
Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022
arXiv 2022
-
[8]
Jake Bruce, Michael D. Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning, 2024
work page 2024
Show all 80 references
-
[9]
Univla: Learning to act anywhere with task-centric latent actions.arXiv preprint arXiv:2505.06111, 2025
Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions.arXiv preprint arXiv:2505.06111, 2025
2025 arXiv
-
[10]
Egocentric-100k, 2025
Build AI. Egocentric-100k, 2025. URLhttps://huggingface.co/datasets/builddotai/Egocentric-100K
2025
-
[11]
Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.IEEE transactions on robotics, 37(6): 1874–1890, 2021
Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.IEEE transactions on robotics, 37(6): 1874–1890, 2021
2021
-
[12]
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021
2021
-
[13]
Worldvla: Towards autoregressive action world model.arXiv preprint arXiv:2506.21539, 2025
Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, et al. Worldvla: Towards autoregressive action world model.arXiv preprint arXiv:2506.21539, 2025. 20
2025 arXiv
-
[14]
Mirage: Cross-embodiment zero-shot policy transfer with cross-painting.arXiv preprint arXiv:2402.19249, 2024
Lawrence Yunliang Chen, Kush Hari, Karthik Dharmarajan, Chenfeng Xu, Quan Vuong, and Ken Goldberg. Mirage: Cross-embodiment zero-shot policy transfer with cross-painting.arXiv preprint arXiv:2402.19249, 2024
2024 arXiv
-
[15]
Moto: Latent motion token as the bridging language for learning robot manipulation from videos
Yi Chen, Yuying Ge, Weiliang Tang, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for learning robot manipulation from videos. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1975...
2025
-
[16]
Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.International Journal of Computer Vision, 130:33–55, 2022
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.Internationa...
2022 doi
-
[17]
Video language planning
Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, Leslie Kaelbling, et al. Video language planning. InInternational Conference on Learning Representations, volume 2024, pages 31138–31155, 2024
2024
-
[18]
Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K. R. Zentner, George Kurian, Sun...
2026 arXiv
-
[19]
Ego4d: Around the world in 3,000 hours of egocentric video
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...
2022
-
[20]
Ego-exo4d: Un- derstanding skilled human activity from first- and third-person perspectives
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, et al. Ego-exo4d: Un- derstanding skilled human activity from first- and third-person perspectives. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1938...
2024
-
[21]
Prediction with action: Visual policy learning via joint denoising process
Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. Advances in Neural Information Processing Systems, 37:112386–112410, 2024
2024
-
[22]
Mastering diverse domains through world models
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023
2023 arXiv
-
[23]
Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025
Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025
2025 arXiv
-
[24]
Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence
Chengkai Hou, Kun Wu, Jiaming Liu, Zhengping Che, Di Wu, Fei Liao, Guangrun Li, Jingyang He, Qiuxuan Feng, Zhao Jin, et al. Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence. arXiv preprint arXiv:2512.24653, 2025. URLhttps...
2025
-
[25]
Video prediction policy: A generalist robot policy with predictive visual representations
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.14803, 2024
2024 arXiv
-
[26]
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin Le...
2025 arXiv
-
[27]
Interndata-m1
InternData-M1 Contributors. Interndata-m1. Hugging Face dataset, 2025. URL https://huggingface.co/ datasets/InternRobotics/InternData-M1
2025
-
[29]
Openvla: An open-source vision-language-action model, 2024
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla...
2024 arXiv
-
[30]
Cosmos policy: Fine-tuning video models for visuomotor control and planning
Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163, 2026
2026 arXiv
-
[31]
Learning to act from actionless videos through dense correspondences
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. InInternational Conference on Learning Representations, volume 2024, pages 40938–40958, 2024
2024
-
[32]
Learning from the best: Smoothness-driven metrics for data quality in imitation learning.arXiv preprint arXiv:2604.23000, 2026
Soham Kulkarni, Raayan Dhar, and Yuchen Cui. Learning from the best: Smoothness-driven metrics for data quality in imitation learning.arXiv preprint arXiv:2604.23000, 2026
2026 arXiv
-
[33]
Let-base-dataset
Leju Robotics. Let-base-dataset. Hugging Face dataset, 2025. URL https://huggingface.co/datasets/ LejuRobotics/LET-Base-Dataset
2025
-
[34]
Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. InProceedings of ...
2023
-
[35]
Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026
Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026
2026 arXiv
-
[36]
Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024
2024 arXiv
-
[37]
Hamster: Hierarchical action models for open-world robot manipulation
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Caelan Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, et al. Hamster: Hierarchical action models for open-world robot manipulation. InInternational Conference on Learning Representations, volume 2025, pages...
2025
-
[38]
Egolive: A large-scale egocentric dataset from real-world human tasks.arXiv preprint arXiv:2604.23570, 2026
Yihang Li, Xuelong Wei, Jingzhou Luo, Yingjing Xiao, Yibo Bai, Guangyuan Zhou, Teng Zou, Chenguang Gui, Jiajun Wen, He Zhang, et al. Egolive: A large-scale egocentric dataset from real-world human tasks.arXiv preprint arXiv:2604.23570, 2026
2026 arXiv
-
[39]
Clam: Continuous latent action models for robot learning from unlabeled demonstrations.arXiv preprint arXiv:2505.04999, 2025
Anthony Liang, Pavel Czempin, Matthew Hong, Yutai Zhou, Erdem Biyik, and Stephen Tu. Clam: Continuous latent action models for robot learning from unlabeled demonstrations.arXiv preprint arXiv:2505.04999, 2025
2025 arXiv
-
[40]
Flow matching for generative modeling
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022
2022 arXiv
-
[41]
Lara: Latent action representation alignment for vision-language-action models.arXiv preprint arXiv:2606.07100, 2026
Mengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang, and Siyuan Huang. Lara: Latent action representation alignment for vision-language-action models.arXiv preprint arXiv:2606.07100, 2026
2026 arXiv
-
[42]
Lda-1b: Scaling latent dynamics action model via universal embodied data ingestion
Jiangran Lyu, Kai Liu, Xuheng Zhang, Haoran Liao, Yusen Feng, Wenxuan Zhu, Tingrui Shen, Jiayi Chen, Jiazhao Zhang, Yifei Dong, et al. Lda-1b: Scaling latent dynamics action model via universal embodied data ingestion. arXiv preprint arXiv:2602.12215, 2026
2026 arXiv
-
[43]
Unifying perception and action: A hybrid- modality pipeline with implicit visual chain-of-thought for robotic action generation
Xiangkai Ma, Lekai Xing, Han Zhang, Wenzhong Li, and Sanglu Lu. Unifying perception and action: A hybrid- modality pipeline with implicit visual chain-of-thought for robotic action generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...
2026
-
[44]
Latent action learning requires supervision in the presence of distractors.arXiv preprint arXiv:2502.00379, 2025
Alexander Nikulin, Ilya Zisman, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, and Vladislav Kurenkov. Latent action learning requires supervision in the presence of distractors.arXiv preprint arXiv:2502.00379, 2025
2025 arXiv
-
[45]
Cosmos-embed1: A joint video-text embedder for physical ai, 2025
NVIDIA, Francesco Ferroni, Prithvijit Chattopadhyay, Greg Heinrich, Mike Ranzinger, Roberto Amoroso, Alice Luo, Andrew Wang, and Ming-Yu Liu. Cosmos-embed1: A joint video-text embedder for physical ai, 2025. URL https://research.nvidia.com/labs/cosmos-lab/cosmos-embed1/
2025
-
[46]
Baihu dataset
OpenLoong. Baihu dataset. AtomGit dataset, 2026. URLhttps://ai.atomgit.com/openloong/baihu-dataset. 22
2026
-
[47]
mimic-video: Video-action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692, 2025
Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, and Elvis Nava. mimic-video: Video-action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692, 2025
2025 arXiv
-
[48]
Reconstructing hands in 3d with transformers
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024
2024
-
[49]
Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025
2025 arXiv
-
[50]
Egoverse: An egocentric human dataset for robot learning from around the world.arXiv preprint arXiv:2604.07607, 2026
Ryan Punamiya, Simar Kareer, Zeyi Liu, Josh Citron, Ri-Zhao Qiu, et al. Egoverse: An egocentric human dataset for robot learning from around the world.arXiv preprint arXiv:2604.07607, 2026. URL https://arxiv.org/abs/ 2604.07607
2026 arXiv
-
[51]
Embodied hands: Modeling and capturing hands and bodies together
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022
2022 arXiv
-
[52]
Xperience-10m: A large-scale egocentric multimodal dataset with structured 3d/4d annotations, 2026
Ropedia. Xperience-10m: A large-scale egocentric multimodal dataset with structured 3d/4d annotations, 2026. URLhttps://huggingface.co/datasets/ropedia-ai/xperience-10m
2026
-
[53]
An efficient metric for data quality measurement in imitation learning
Noushad Sojib and Momotaz Begum. An efficient metric for data quality measurement in imitation learning. arXiv preprint arXiv:2605.01544, 2026
2026 arXiv
-
[54]
Emma-x: An embodied multimodal action model with grounded chain of thought and look-ahead spatial reasoning
Qi Sun, Pengfei Hong, Tej Deep Pala, Vernon Toh, U-Xuan Tan, Deepanway Ghosal, and Soujanya Poria. Emma-x: An embodied multimodal action model with grounded chain of thought and look-ahead spatial reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Comp...
2025
-
[55]
Galaxea g0.5 technical report.arxiv, 2026
Galaxea Team. Galaxea g0.5 technical report.arxiv, 2026. URL https://opengalaxea.github.io/G05/Galaxea_ G0_5.pdf
2026
-
[56]
Motubrain: An advanced world action model for robot control.arXiv preprint arXiv:2604.27792, 2026
MotuBrain Team, Chendong Xiang, Fan Bao, Haitian Liu, Hengkai Tan, Hongzhe Bi, James Li, Jiabao Liu, Jingrui Pang, Kiro Jing, et al. Motubrain: An advanced world action model for robot control.arXiv preprint arXiv:2604.27792, 2026
2026 arXiv
-
[57]
Predictive in- verse dynamics models are scalable learners for robotic manipulation
Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive in- verse dynamics models are scalable learners for robotic manipulation. InInternational Conference on Learning Representations, volume 2025, pages 92033–92052, 2025
2025
-
[58]
Interndata-a1: Pioneering high-fidelity synthetic data for pre-training generalist policy.arXiv preprint arXiv:2511.16651, 2025
Yang Tian, Yuyin Yang, Yiman Xie, Zetao Cai, Xu Shi, Ning Gao, Hangxu Liu, Xuekun Jiang, Zherui Qiu, Feng Yuan, et al. Interndata-a1: Pioneering high-fidelity synthetic data for pre-training generalist policy.arXiv preprint arXiv:2511.16651, 2025. URLhttps://arxiv.org/abs/2511.16651
2025
-
[59]
Vggt: Visual geometry grounded transformer
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5294–5306, 2025
2025
-
[60]
Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers
Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers. Advances in neural information processing systems, 37:124420–124450, 2024
2024
-
[61]
Vq-vla: Improving vision-language-action models via scaling vector-quantized action tokenizers
Yating Wang, Haoyi Zhu, Mingyu Liu, Jiange Yang, Hao-Shu Fang, and Tong He. Vq-vla: Improving vision-language-action models via scaling vector-quantized action tokenizers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11089–11099, 2025
2025
-
[62]
Humanego: Zero-shot robot learning from minutes of human egocentric videos.arXiv e-prints, pages arXiv–2605, 2026
Zhi Wang, Botao He, Kelin Yu, Seungjae Lee, Ruohan Gao, Furong Huang, and Yiannis Aloimonos. Humanego: Zero-shot robot learning from minutes of human egocentric videos.arXiv e-prints, pages arXiv–2605, 2026
2026
-
[63]
Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation
Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024. URLhttps://arxiv.or...
2024 arXiv
-
[64]
Robocoin: An open-sourced bimanual robotic data collection for integrated manipulation
Shihan Wu, Xuecheng Liu, Shaoxuan Xie, Pengwei Wang, Xinghang Li, Bowen Yang, Zhe Li, Kai Zhu, Hongyu Wu, Yiheng Liu, et al. Robocoin: An open-sourced bimanual robotic data collection for integrated manipulation. arXiv preprint arXiv:2511.17441, 2025. URLhttps://arxiv.org/abs/...
2025 arXiv
-
[65]
Rl token: Bootstrapping online rl with vision-language-action models.arXiv preprint arXiv:2604.23073, 2026
Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, and Liyiming Ke. Rl token: Bootstrapping online rl with vision-language-action models.arXiv preprint arXiv:2604.23073, 2026
2026 arXiv
-
[66]
Gigaworld-policy: An efficient action-centered world–action model.arXiv preprint arXiv:2603.17240, 2026
Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jie Li, Jindi Lv, Jingyu Liu, et al. Gigaworld-policy: An efficient action-centered world–action model.arXiv preprint arXiv:2603.17240, 2026
2026
-
[67]
Latent action pretraining from videos
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. InInternational Conference on Learning Representations, volume 2025, pages 28213–28239, 2025
2025
-
[68]
World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026
Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026
2026 arXiv
-
[69]
Genie sim 3.0: A high-fidelity comprehensive simulation platform for humanoid robot.arXiv preprint arXiv:2601.02078, 2026
Chenghao Yin, Da Huang, Di Yang, Jichao Wang, Nanshu Zhao, Chen Xu, Wenjun Sun, Linjie Hou, Zhijun Li, Junhui Wu, et al. Genie sim 3.0: A high-fidelity comprehensive simulation platform for humanoid robot.arXiv preprint arXiv:2601.02078, 2026. URLhttps://arxiv.org/abs/2601.02078
2026 arXiv
-
[70]
Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models
Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, Jiazhao Zhang, et al. Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models. arXiv preprint arXiv:2606.17846, 2026
2026 arXiv
-
[71]
Fast-wam: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026
Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026
2026 arXiv
-
[72]
Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024
2024 arXiv
-
[73]
Atomicvla: Unlocking the potential of atomic skill learning in robots, 2026
Likui Zhang, Tao Tang, Zhihao Zhan, Xiuwei Chen, Zisheng Chen, Jianhua Han, Jiangtong Zhu, Pei Xu, Hang Xu, Hefeng Wu, Liang Lin, and Xiaodan Liang. Atomicvla: Unlocking the potential of atomic skill learning in robots, 2026. URLhttps://arxiv.org/abs/2603.07648
2026
-
[74]
Efros, Eli Shechtman, and Oliver Wang
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018
2018
-
[75]
Joyai-ra 0.1: A foundation model for robotic autonomy.arXiv preprintarXiv:2604.20100, 2026
Tianle Zhang, Zhihao Yuan, Dafeng Chi, Peidong Liu, Dongwei Li, Kejun Hu, Likui Zhang, Junnan Nie, Ziming Wei, Zengjue Chen, et al. Joyai-ra 0.1: A foundation model for robotic autonomy.arXiv preprintarXiv:2604.20100, 2026
2026 arXiv
-
[76]
Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge
Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, He Wang, Zhizheng Zhang, et al. Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge. Advances in Neural Information Processing Systems, 38:24195...
2026
-
[77]
Cot-vla: Visual chain-of-thought reasoning for vision-language-action models
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Confere...
2025
-
[78]
Sim2real vla: Zero-shot generalization of synthesized skills to realistic manipulation
Runyi Zhao, Sheng Xu, Ruixing Jin, Yueci Deng, Yunxin Tai, Kui Jia, and Guiliang Liu. Sim2real vla: Zero-shot generalization of synthesized skills to realistic manipulation. InThe Fourteenth International Conference on Learning Representations, 2026
2026
-
[79]
Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets.arXiv preprint arXiv:2504.02792, 2025
Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets.arXiv preprint arXiv:2504.02792, 2025
2025 arXiv
-
[80]
Rt-2: Vision-language-action models transfer web knowledge to robotic control
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 24 Ap...
2023
-
[2025]
URLhttps://arxiv.org/abs/2509.00576. 21
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.