Pith. sign in

REVIEW 4 major objections 5 minor 117 references

Robots Acquire Manipulation Skills in Seconds from a Single Human Video

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HOST lets a robot learn a novel manipulation skill from a single human video in about 29 seconds, at 62% average success with no parameter updates, while retaining mastered skills.

desk verdict Strong system contribution, well-executed study; the headline margin over fine-tuning is not statistically established, but the core inference-time claim is credible. read the letter →

arxiv 2607.20033 v3 pith:DDDLMOTV submitted 2026-07-22 cs.RO

Guangyan Chen , Meiling Wang , Te Cui , Zichen Zhou , Qi Shao , Shalfun Li , Hang Su , Roy Gan
show 4 more authors
Hao Wang Mengyin Fu Yi Yang Yufeng Yue
This is my paper
classification cs.RO
keywords one-shotvisualimitationinference-timeskillacquisitionhumanvideodemonstrationtaskprogressalignmentself-groundedpredictionretentionbimanualmanipulationautoregressiveflowmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a robot can acquire a new manipulation skill at inference time from a single human video, with no gradient updates, in about 29 seconds, while keeping skills it already knows. The method, HOST, reaches 62% average success on 50 held-out manipulation tasks, outperforms zero-shot and one-shot visual-imitation baselines by 43–45 percentage points, and exceeds a strong baseline fine-tuned on 50 robot demonstrations per task while using 50 times fewer demonstrations and acquiring each skill 507 times faster. The reason this matters is that current skill acquisition requires collecting teleoperated demonstrations and hours of fine-tuning, which also erodes previously mastered skills. HOST instead treats the human video as an external specification that drives a cascade of prediction: where the robot is in the demonstrated procedure, what the robot's own next observations will look like, and what actions realize them.

What carries the argument

The load-bearing object is a learned task-progress manifold with a monotonic frame correspondence between the human video and robot trajectories, recovered self-supervised via Smooth Dynamic Time Warping and temporal cycle-consistency. This correspondence redefines the robot's prediction target at each step, making the video the active driver of the prediction instead of passive context. Execution is then resolved by a causal cascade inside a single autoregressive diffusion transformer: a localization token estimates where the robot is in the video window, future-observation tokens translate the video's upcoming progression into the robot's own camera views, and action tokens decode those pr

What would settle it

Run the trained HOST model, without any retraining, on a single-arm robot or a humanoid with different kinematics and camera views, giving it the same held-out human videos; if success on novel tasks falls well below the reported 62% average, the cross-embodiment claim fails. A cheaper in-domain check is to compare the localization head's predicted progress against human-annotated progress on videos with variable speed, repeated steps, or ambiguous sub-goals.

Watch

Extended reading notes

Core claim

The discovery is that the structural mismatch between a human video and robot execution—different timing, viewpoint, embodiment, and appearance—can be resolved by coupling the prediction target to the video's future progression and then letting the policy generate its own future observations before actions. On a shared task-progress manifold, the robot and the human video are aligned frame-to-frame without manual annotation. At each robot timestep, the target is not a fixed temporal offset but the segment of the robot's own future trajectory that corresponds to the upcoming segment of the video. A frozen autoregressive flow-matching policy then outputs, in order, a localization scalar, futur

Load-bearing premise

The method is evaluated on a single two-armed robot; if it cannot transfer to robots with substantially different bodies and grippers, the broad claim of learning manipulation skills from one human video is not yet established.

Editorial extensions

If this is right

  • A human video becomes a complete skill specification: acquiring a new skill takes roughly the time of recording the video (~29 s), not hours of teleoperation and fine-tuning.
  • Because policy weights stay frozen during acquisition, previously mastered skills are not overwritten; the paper measures retention near 100%, versus 20–43% for fine-tuned baselines.
  • Stored videos can be retrieved and reused for recurring tasks, so skills accumulate as an external library without repeated human involvement.
  • The specific design choices—progress-based target coupling and the self-grounded cascade—are each shown to be necessary: ablations drop success from 62% to 45% without progress-based windowing and to 34% when actions are predicted directly.
  • If the central claim holds, robotic deployment can shift from 'collect data and retrain per task' to 'record one video and execute,' which is the practical promise of one-shot visual imitation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference — If cross-embodiment transfer proves weak, the same two-stage recipe could be retrained with human–robot pairs from other robot bodies; the paper explicitly leaves this untested, so the breadth of the claim rests on it.
  • Editorial inference — The causal predict-your-own-observations step suggests a diagnostic: on tasks with repeated or easily confused stages, the localization head should err first; measuring its predicted progress against human annotations on such tasks would isolate where HOST breaks.
  • Editorial inference — Since HOST stores skills as videos and retrieves them by instruction/scene similarity, its long-horizon limit is likely task discrimination, not execution; the paper's own discussion anticipates this as memory grows.
  • Editorial inference — If reproduced on other platforms, HOST's numbers imply that a small Stage-1 robot corpus plus a modest human-video set could be an economical route to 'watch once and do' policies, at the price of a large up-front pretraining cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces HOST, a framework that acquires manipulation skills from a single human video at inference time without parameter updates. The method couples prediction targets to the future progression of the video demonstration through a self-supervised temporal alignment module (SDTW + TCC) that maps robot trajectories and videos to a shared task-progress manifold, then resolves execution through a causal cascade: progress localization, future-observation prediction, and action prediction in an autoregressive flow-matching transformer. Training is two-stage: large-scale same-embodiment robot-robot pretraining (193,462 trajectories, 229 tasks) followed by adaptation with 5,847 human-robot pairs. On a bimanual ARX R5 platform, the authors report 62% success across 50 novel tasks from a single video, 43-45% gains over OSVI/zero-shot baselines, a 6-point margin over the best 50-demonstration fine-tuned baseline, retention of previously mastered skills, robustness to perturbations, ablations of each mechanism, and a retrieval-based persistence mechanism.

Significance. If sustained, HOST would be a significant step toward practical one-shot visual imitation: it replaces a collection-and-fine-tuning loop with an inference-time video condition, demonstrates large experimental breadth (50 novel tasks, 20 trials each, human evaluators), and provides ablations showing that target coupling and self-grounded prediction each contribute. The two-stage training scheme is a thoughtful response to the scarcity of human-robot paired data, and the persistence/retrieval module addresses a practical deployment concern. The strengths are real: the evaluation is substantial relative to much of the OSVI literature, and the large margins over zero-shot and OSVI baselines are not the main point of concern. However, the statistical support for one headline comparative claim and the internal validity of the localization metric require attention before the paper can be accepted.

major comments (4)
  1. [Sec. 2.5, Figs. 6A/C and Abstract] The claim that HOST 'even exceeds the baseline fine-tuned on 50 robot demonstrations per task' rests on a 62% vs 56% margin over Wall-OSS+SFT. No confidence intervals or significance tests are reported. If the comparison is on the eight tasks of Fig. 5 (as Fig. 6C suggests), n=160 per method and the pooled two-proportion standard error is approximately 5.5 points, so the 6-point margin is roughly one standard error (per-task clustering widens it further). The authors should report per-task success counts, CIs, and an appropriate paired/clustered test for this comparison, or soften the claim. The paper's own per-task qualifier ('this advantage holds across most of these tasks') concedes losses on some tasks, which is consistent with a noise-level margin.
  2. [Sec. 2.5 / Fig. 6 versus Sec. 2.3] The 62% average is defined over the full 50-task set (Sec. 2.3), while the fine-tuning comparison appears to use a smaller subset (the eight tasks of Fig. 5). The abstract and Sec. 2.1 present the fine-tuning comparison as if it applied to the full benchmark. Please state explicitly how many tasks are used in Fig. 6A/C, list them, and either provide full 50-task fine-tuning results or clearly scope the 'exceeds fine-tuned on 50 demos' claim to the subset. Without this, the headline comparison mixes two different evaluation sets.
  3. [Sec. 2.9, Eqs. (6)-(7), Fig. 10B] The localization accuracy evaluation uses 'ground-truth progress values provided by the alignment module' - the same module that constructs the training targets in Sec. 4.2. This is circular for an accuracy claim: it measures whether the localization head reproduces the alignment module's output, not whether the progress estimate is correct against true task progress. The manuscript already has an independent human-annotation protocol in Sec. 2.8 (Fig. 9D); apply that protocol to the localization head, or relabel the metric as 'alignment-module consistency' and discuss the interpretation accordingly.
  4. [Discussion / Sec. 2.2] All experiments use a single bimanual platform (ARX R5, 20-DoF); the Discussion correctly acknowledges that HOST 'remains untested on robots that differ substantially in embodiment.' Given that the title and abstract promise human-video skill acquisition generally, the cross-embodiment claim should be scoped to the tested platform, or the paper should include at least one additional embodiment or joint-level transfer check. This is a scope limitation rather than an internal inconsistency, but it is load-bearing for the generality of the central claim.
minor comments (5)
  1. [Sec. 2.4] Define 'zero-shot' as language-conditioned zero-shot and distinguish it from one-shot visual imitation; the 45% and 43% margins are easy to conflate.
  2. [Eq. 12] The '||' notation for token-sequence concatenation is not defined; add a sentence.
  3. [Abstract] The abstract mentions 'project website and GitHub repository' but no URLs appear in the text; include them.
  4. [Fig. 10E] Report retrieval accuracy and novelty recognition with confidence intervals or trial counts; the threshold-band claim would be easier to evaluate with error bars.
  5. [Sec. 2.9] Use 'alignment-derived labels' rather than 'ground-truth progress' when the alignment module's output is the reference.

Circularity Check

1 steps flagged · score 3.0 of 10

One supporting metric (localization accuracy) is self-referential, but the central skill-acquisition claims are externally benchmarked and do not reduce to the method's inputs.

  1. fitted input called prediction [Sec. 2.9 (Fig. 10B), with labels/targets defined in Sec. 4.2 (Eq. 7) and trained by Eq. 15]
    "We evaluate localization accuracy on the novel tasks, using ground-truth progress values provided by the alignment module. The predicted progress p̂_t closely tracks the ground-truth p_t across all steps, as presented in Fig. 10B, with a mean absolute error of 0.013 in normalized progress."

    The 'ground-truth' progress values are produced by the same alignment module that constructs the scalar localization labels p_t and the coupled prediction targets T_t used to train the policy (Sec. 4.2: 'The current progress of the robot within W is recorded as a scalar localization label p_t'; Eq. 15 trains L_loc on y_p = E_p(p_t)). The reported 0.013 MAE therefore measures how well the localization head reproduces its own training-label generator, not agreement with an independent ground truth. This is a fitted-input-called-prediction loop. It is a supporting internal metric rather than the core claim: the alignment module itself is checked against human annotations in Fig. 9D, and the headline success/retention results are judged by external task success.

full rationale

The core derivation is an empirical system-building story, not a mathematical derivation. The headline claims—62% success on 50 novel tasks, 43–45% margins over OSVI/zero-shot baselines, and retention—are measured by externally judged task success, so they cannot reduce to the method's inputs by construction. The two mechanisms (target coupling and self-grounded prediction) are supported by ablations on held-out tasks, and the equations define a training objective rather than a self-referential prediction. The one genuinely circular evaluation I found is in Sec. 2.9: localization accuracy is scored against 'ground-truth progress values provided by the alignment module,' i.e., the same module that generated the p_t labels and coupled targets used to supervise the localization head. That makes the 0.013 MAE a fit-quality statistic with respect to the system's own label generator. This is not load-bearing for the central result because alignment is independently validated against human-annotated events (Fig. 9D) and the main success rates are externally judged. Separately, the 6-point margin over the best SFT baseline (62% vs 56%) lacks error bars or significance testing, which is a statistical robustness concern, not a circularity concern. On balance, the circular content is limited to a secondary self-referential metric, so the score is 3 rather than 0.

Assumptions & free parameters 8 free parameters · 5 assumptions · 1 invented entities

The method's behavior hinges on a handful of tuned alignment and loss hyperparameters, and on the domain assumption that monotonic task-progress alignment between human video and robot execution is learnable from paired data on the same platform. The invented entity is a latent alignment manifold, which has partial external validation. No new physical entities or constants are introduced.

free parameters (8)
  • SDTW softmin temperature γ = 1.0
    Controls soft-min sharpness in alignment (Supp. Table 1). Chosen by the authors; affects matching probabilities and hence all coupled targets.
  • Column-normalization temperature γ_f = 0.1
    Sharpens cost matrix in Eq. 2 (Supp. Table 1). Directly shapes the alignment used to construct training targets.
  • Softmax temperature κ = 0.1
    Scales pairwise similarity in Eq. 2 (Supp. Table 1). Tuned, affects alignment quality.
  • TCC variance weight λ_var / DTW loss weight λ_DTW = 0.001 / 0.3
    Balance alignment losses in Eq. 9-10 (Supp. Table 1). Chosen by hand; if changed, alignment and therefore targets change.
  • Policy loss weights λ_o, λ_a, λ_p = 1.0 / 10.0 / 1.0
    Balance observation, action, and localization objectives in Eq. 16 (Supp. Table 3). Tuned; action weight is 10x to compensate token count.
  • NCP noise scale / probability = 0.5 / 0.5
    Noise added to clean progress token during training (Supp. B.2). Tuned for robust autoregressive conditioning.
  • Retrieval threshold δ★ and mixing weight ω = δ★ (exact value not stated); ω in Eq. 20
    Determines when a stored video is reused vs. a new video is requested (Sec. 2.11). The paper says δ★ lies within a wide band but does not give its value or ω.
  • Video window length L / action chunk H / downsample ratio = 192 frames / 32 steps / 8:1
    Design choices controlling context and target horizon (Supp. Table 4). Not derived from any principle.
assumptions (5)
  • domain assumption A shared task-progress manifold exists that monotonically aligns any two executions of the same manipulation task.
    The entire target-coupling mechanism (Sec. 4.2, Eq. 3-7) assumes that Smooth DTW + temporal cycle-consistency can recover frame-level correspondence between a human video and a robot trajectory. This is empirically validated against human annotations in Fig. 9D, but remains an assumption for novel tasks and novel embodiments.
  • domain assumption Task progress is a scalar, monotonic quantity across both video demonstration and robot trajectory.
    The localization label p_t and window advancement Eq. 18 rely on strictly monotonic progress. Boundary clipping and monotonic path constraints in Eq. 3 bake in this assumption; tasks with loops or optional sub-steps would break it.
  • domain assumption Same-embodiment robot–robot pretraining transfers to human-video conditioning with only 5,847 human–robot pairs.
    The two-stage protocol (Sec. 4.4) assumes that Stage 1 builds a generic 'follow-a-demonstration' capability that Stage 2 can adapt from robot videos to human videos using a comparatively small paired set. This is plausible and ablated (Fig. 10D) but only on one platform.
  • domain assumption Human evaluators' success judgments against task-specific completion criteria are reliable and unbiased.
    All headline success rates depend on human judges (Sec. 2.2); no inter-evaluator agreement or blind evaluation protocol is reported.
  • ad hoc to paper The 50 novel tasks selected are representative of the manipulation skills the method claims to acquire.
    The task set is self-selected and spans varied objects/tools/primitives (Sec. 2.3), but no criteria are given for how they were chosen or whether failures on easier tasks were excluded.
invented entities (1)
  • Shared task progress manifold (learned embedding space for aligning human video and robot trajectory frames) independent evidence
    purpose: Provides frame-level correspondence for target coupling and window advancement (Sec. 4.2).
    The manifold is validated against human-annotated events in Fig. 9D (MAD 0.006 vs 0.079 for clock-time), giving it an external falsifiable handle within the paper. It is still a learned latent construct, not a physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robots Acquire Manipulation Skills in Seconds from a Single Human Video." pith.science (2026). https://pith.science/paper/DDDLMOTV

@misc{pith2026260720033,
  author       = {Pith},
  title        = {Pith review of: Robots Acquire Manipulation Skills in Seconds from a Single Human Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDDLMOTV}},
  note         = {Machine review of arXiv:2607.20033}
}
read the original abstract

The ability to acquire skills rapidly and effortlessly while retaining those already mastered is essential for robots. However, current methods still rely on a cumbersome training-time loop that is costly and slow, while eroding skills already mastered. In this paper, we introduce HOST (Human-to-robot One-Shot Skill AcquisiTion), a framework that enables a robot to acquire skills in seconds from a single human video while retaining previously mastered skills. HOST resolves skill acquisition through a cascade of self-grounded prediction. It first estimates the robot's progress within the demonstrated task, then translates the upcoming progression into the robot's own future observations, and finally derives actions from these predicted observations. This cascade is trained on targets coupled to the video demonstration, obtained by mapping the robot trajectory and the video demonstration onto a shared task progress manifold, then redefining each target to align with the future progression of the video. HOST thereby enables the robot to actively follow the demonstrated procedure and adapt it to the robot's embodiment. HOST acquires novel skills at inference time from a single human video in an average of 29 seconds and achieves a 62% average success rate. It exceeds the zero-shot baseline by 45% while retaining previously mastered skills. HOST even exceeds the baseline fine-tuned on 50 robot demonstrations per task while requiring 50 times fewer demonstrations and acquiring each skill 507 times faster. Additional information about HOST is available on the project website.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

117 extracted references · 7 canonical work pages

  1. [1]

    Learning agile and dynamic motor skills for legged robots.Science Robotics, 4(26):eaau5872, 2019

    Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots.Science Robotics, 4(26):eaau5872, 2019

    2019

  2. [2]

    Learning quadrupedal locomotion over challenging terrain.Science Robotics, 5(47):eabc5986, 2020

    Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain.Science Robotics, 5(47):eabc5986, 2020

    2020

  3. [3]

    Visual dexterity: In-hand reorientation of novel and complex object shapes.Science Robotics, 8(84):eadc9244, 2023

    Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes.Science Robotics, 8(84):eadc9244, 2023

    2023

  4. [4]

    Solving Rubik’s cube with a robot hand

    OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving Rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019

    arXiv 1910

  5. [5]

    QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. InConference on Robot Learning, volume 87, pages 651–673. PMLR, 2018

    2018

  6. [6]

    Scaling up multi-task robotic reinforcement learning

    Dmitry Kalashnikov, Jake Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Scaling up multi-task robotic reinforcement learning. InProceedings of the 5th Conference on Robot Learning, volume 164 ofProceedings of Machine Learning Research, pages 557–575. PMLR,

  7. [7]

    A generalized path integral control approach to reinforce- ment learning.Journal of Machine Learning Research, 11:3137–3181, 2010

    Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforce- ment learning.Journal of Machine Learning Research, 11:3137–3181, 2010

    2010

  8. [8]

    Path integral guided policy search

    Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3381–3388. IEEE,

Show all 117 references
  1. [9]

    Learning coordinated badminton skills for legged manipulators.Science Robotics, 10(102):eadu3922, 2025

    Yuntao Ma, Andrei Cramariuc, Farbod Farshidian, and Marco Hutter. Learning coordinated badminton skills for legged manipulators.Science Robotics, 10(102):eadu3922, 2025. doi: 10.1126/scirobotics.adu3922

    2025 doi

  2. [10]

    Precise and dexterous robotic manipulation via human-in- the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in- the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025. doi: 10.1126/scirobotics.ads5033

    2025 doi

  3. [11]

    Barreiros, Aykut Özgün Önol, Mengchao Zhang, Sam Creasey, Aimee Goncalves, Andrew Beaulieu, Aditya Bhat, Kate M

    Jose A. Barreiros, Aykut Özgün Önol, Mengchao Zhang, Sam Creasey, Aimee Goncalves, Andrew Beaulieu, Aditya Bhat, Kate M. Tsui, and Alex Alspach. Learning contact-rich whole-body manipulation with example-guided reinforcement learning.Science Robotics, 10(105):eads6790, 2025. d...

    2025 doi

  4. [12]

    Is imitation learning the route to humanoid robots?Trends in Cognitive Sciences, 3(6):233–242, 1999

    Stefan Schaal. Is imitation learning the route to humanoid robots?Trends in Cognitive Sciences, 3(6):233–242, 1999

    1999

  5. [13]

    Sanketi, Grecia Salazar, Michael S

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryo...

    2023

  6. [14]

    In Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, 6 2025

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang S...

    2025

  7. [15]

    In9th Annual Conference on Robot Learning, 2025

    Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y Galliker, et al.𝜋0.5: A vision-language-action model with open-world generalization. In9th Annual Conference on Robot Learning, 2025...

    2025 arXiv

  8. [16]

    Toward next-generation learned robot manipulation.Science Robotics, 6(54):eabd9461,

    Jinda Cui and Jeff Trinkle. Toward next-generation learned robot manipulation.Science Robotics, 6(54):eabd9461,

  9. [17]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  10. [18]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  11. [19]

    RT-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  12. [20]

    Open X-Embodiment: Robotic learning datasets and RT-X models

    Open X-Embodiment Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X models. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024. doi: 10.1109/ ICRA57147.2024.10611477

    2024

  13. [21]

    Octo: An open-source generalist robot policy

    Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-...

    2024 doi

  14. [22]

    From intention to execution: Probing the generaliza- tion boundaries of vision-language-action models.arXiv preprint arXiv:2506.09930, 2025

    Irving Fang, Juexiao Zhang, Shengbang Tong, and Chen Feng. From intention to execution: Probing the generaliza- tion boundaries of vision-language-action models.arXiv preprint arXiv:2506.09930, 2025

    2025 arXiv

  15. [23]

    Ricl: Adding in-context adaptability to pre-trained vision-language-action models

    Kaustubh Sridhar et al. Ricl: Adding in-context adaptability to pre-trained vision-language-action models. In Conference on Robot Learning, 2025

    2025

  16. [24]

    FLaRe: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning

    Jiaheng Hu, Rose Hendrix, Ali Farhadi, Aniruddha Kembhavi, Roberto Martin-Martin, Peter Stone, Kuo-Hao Zeng, and Kiana Ehsani. FLaRe: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning. In2025 IEEE International Conference on Ro...

    2025

  17. [25]

    Qingwei Dong, Peng Zeng, Yunpeng He, Guangxi Wan, and Xiaoting Dong. Mitigating catastrophic forgetting in robot continual learning: A guided policy search approach enhanced with memory-aware synapses.IEEE Robotics and Automation Letters, 9(12):11242–11249, 2024. doi: 10.1109/...

    2024

  18. [26]

    Learning by watching: Extracting reusable task knowledge from visual observation of human performance.IEEE Transactions on Robotics and Automation, 10(6):799–822, 1994

    Yasuo Kuniyoshi, Masayuki Inaba, and Hirochika Inoue. Learning by watching: Extracting reusable task knowledge from visual observation of human performance.IEEE Transactions on Robotics and Automation, 10(6):799–822, 1994

    1994

  19. [27]

    One-shot visual imitation learning via meta-learning

    Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. InConference on Robot Learning, pages 357–368. PMLR, 2017

    2017

  20. [28]

    One-shot imitation from observing humans via domain-adaptive meta-learning

    Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. InProceedings of Robotics: Science and Systems (RSS), 2018

    2018

  21. [29]

    Towards more generalizable one-shot visual imitation learning

    Zhao Mandi, Fangchen Liu, Kimin Lee, and Pieter Abbeel. Towards more generalizable one-shot visual imitation learning. In2022 IEEE International Conference on Robotics and Automation (ICRA), pages 2434–2444. IEEE, 2022

    2022

  22. [30]

    Vid2Robot: 23 End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers

    Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski, Yonatan Bisk, and Debidatta Dwibedi. Vid2Robot: 23 End-to-end Video-conditioned Policy Learning with Cross-...

    2024 doi

  23. [31]

    Osvi-wm: One-shot visual imitation for unseen tasks using world-model-guided trajectory generation

    Raktim Goswami, Prashanth Krishnamurthy, Yann LeCun, and Farshad Khorrami. Osvi-wm: One-shot visual imitation for unseen tasks using world-model-guided trajectory generation. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances i...

    2025

  24. [32]

    Remembering the past to imagine the future: the prospective brain.Nature Reviews Neuroscience, 8(9):657–661, 2007

    Daniel L Schacter, Donna Rose Addis, and Randy L Buckner. Remembering the past to imagine the future: the prospective brain.Nature Reviews Neuroscience, 8(9):657–661, 2007

    2007

  25. [33]

    Infant imitation after a 1-week delay: long-term memory for novel acts and multiple stimuli

    Andrew N Meltzoff. Infant imitation after a 1-week delay: long-term memory for novel acts and multiple stimuli. Developmental Psychology, 24(4):470–476, 1988

    1988

  26. [34]

    Reinforcement learning and episodic memory in humans and animals: an integrative framework.Annual Review of Psychology, 68:101–128, 2017

    Samuel J Gershman and Nathaniel D Daw. Reinforcement learning and episodic memory in humans and animals: an integrative framework.Annual Review of Psychology, 68:101–128, 2017

    2017

  27. [35]

    Neural simulation of action: a unifying mechanism for motor cognition.NeuroImage, 14(1): S103–S109, 2001

    Marc Jeannerod. Neural simulation of action: a unifying mechanism for motor cognition.NeuroImage, 14(1): S103–S109, 2001

    2001

  28. [36]

    A unifying computational framework for motor control and social interaction.Philosophical Transactions of the Royal Society B: Biological Sciences, 358(1431):593–602, 2003

    Daniel M Wolpert, Kenji Doya, and Mitsuo Kawato. A unifying computational framework for motor control and social interaction.Philosophical Transactions of the Royal Society B: Biological Sciences, 358(1431):593–602, 2003

    2003

  29. [37]

    Neurophysiological mechanisms underlying the understanding and imitation of action.Nature Reviews Neuroscience, 2(9):661–670, 2001

    Giacomo Rizzolatti, Leonardo Fogassi, and Vittorio Gallese. Neurophysiological mechanisms underlying the understanding and imitation of action.Nature Reviews Neuroscience, 2(9):661–670, 2001

    2001

  30. [38]

    Neural circuits underlying imitation learning of hand actions: an event-related fmri study.Neuron, 42(2):323–334,

    Giovanni Buccino, Stefan Vogt, Afra Ritzl, Gereon R Fink, Karl Zilles, Hans-Joachim Freund, and Giacomo Rizzolatti. Neural circuits underlying imitation learning of hand actions: an event-related fmri study.Neuron, 42(2):323–334,

  31. [39]

    One-shot visual imitation via attributed waypoints and demonstration augmentation

    Matthew Chang and Saurabh Gupta. One-shot visual imitation via attributed waypoints and demonstration augmentation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5055–5062. IEEE, 2023

    2023

  32. [40]

    Igniting vlms toward the embodied space.arXiv preprint arXiv:2509.11766, 2025

    Andy Zhai, Brae Liu, Bruno Fang, Chalse Cai, Ellie Ma, Ethan Yin, Hao Wang, Hugo Zhou, James Wang, Lights Shi, et al. Igniting vlms toward the embodied space.arXiv preprint arXiv:2509.11766, 2025

    2025

  33. [41]

    Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    2026 arXiv

  34. [42]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. InInternational Conference on Learning Representations, 2022

    2022

  35. [43]

    Tenenbaum, and Alberto Rodriguez

    Nima Fazeli, Miquel Oller, Jiajun Wu, Zheng Wu, Joshua B. Tenenbaum, and Alberto Rodriguez. See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion.Science Robotics, 4(26):eaav3123,

  36. [44]

    Representation learning via global temporal alignment and cycle-consistency

    Isma Hadji, Konstantinos G Derpanis, and Allan D Jepson. Representation learning via global temporal alignment and cycle-consistency. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11068–11077, 2021

    2021

  37. [45]

    Temporal cycle- consistency learning

    Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle- consistency learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1801–1810, 2019

    2019

  38. [46]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  39. [47]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  40. [48]

    Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.Transactions on Machine Learning Research, 2025

    Weixin Liang, Lili Yu, Liang Luo, Srini Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, and Xi Victoria Lin. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.Transactions on Machine Learning R...

    2025

  41. [49]

    Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining

    Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, and Noah Constant. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  42. [50]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe Eleventh International Conference on Learning Representations, 2023. 24

    2023

  43. [51]

    A survey of robot learning from demonstra- tion.Robotics and Autonomous Systems, 57(5):469–483, 2009

    Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstra- tion.Robotics and Autonomous Systems, 57(5):469–483, 2009

    2009

  44. [52]

    Recent advances in robot learning from demonstration.Annual Review of Control, Robotics, and Autonomous Systems, 3:297–330, 2020

    Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration.Annual Review of Control, Robotics, and Autonomous Systems, 3:297–330, 2020

    2020

  45. [53]

    Deep imitation learning for complex manipulation tasks from virtual reality teleoperation

    Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628–5635. IEEE, 2018

    2018

  46. [54]

    White, De Ru Tsai, Richard Jaepyeong Cha, Jeffrey Jopling, Chelsea Finn, and Axel Krieger

    Ji Woong (Brian) Kim, Juo-Tung Chen, Pascal Hansen, Lucy Xiaoyang Shi, Antony Goldenberg, Samuel Schmidgall, Paul Maria Scheikl, Anton Deguet, Brandon M. White, De Ru Tsai, Richard Jaepyeong Cha, Jeffrey Jopling, Chelsea Finn, and Axel Krieger. SRT-H: A hierarchical framework ...

    2025 doi

  47. [55]

    Pomerleau

    Dean A. Pomerleau. ALVINN: An autonomous land vehicle in a neural network. In David S. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988

    1988

  48. [56]

    DexMV: Imitation learning for dexterous manipulation from human videos

    Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. DexMV: Imitation learning for dexterous manipulation from human videos. InEuropean Conference on Computer Vision, pages 570–587. Springer, 2022

    2022

  49. [57]

    DexCap: Scalable and portable mocap data collection system for dexterous manipulation

    Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C Karen Liu. DexCap: Scalable and portable mocap data collection system for dexterous manipulation. InProceedings of Robotics: Science and Systems (RSS), 2024

    2024

  50. [58]

    Time- contrastive networks: Self-supervised learning from video

    Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time- contrastive networks: Self-supervised learning from video. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134–1141. IEEE, 2018

    2018

  51. [59]

    XIRL: Cross-embodiment inverse reinforcement learning

    Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. InConference on Robot Learning, pages 537–546. PMLR, 2022

    2022

  52. [60]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. InConference on Robot Learning, pages 1113–1132. PMLR, 2020

    2020

  53. [61]

    WHIRL: Human-to-robot imitation in the wild

    Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. WHIRL: Human-to-robot imitation in the wild. InProceedings of Robotics: Science and Systems (RSS), 2022

    2022

  54. [62]

    MimicPlay: Long-horizon imitation learning by watching human play

    Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Animashree Anandkumar. MimicPlay: Long-horizon imitation learning by watching human play. InConference on Robot Learning, volume 229, pages 201–221. PMLR, 2023

    2023

  55. [63]

    EgoScale: Scaling dexterous manipulation with diverse egocentric human data.arXiv preprint arXiv:2602.16710, 2026

    Ruijie Zheng, Dantong Niu, Yuqi Xie, Jing Wang, Mengda Xu, Yunfan Jiang, Fernando Castañeda, Fengyuan Hu, You Liang Tan, Letian Fu, Trevor Darrell, Furong Huang, Yuke Zhu, Danfei Xu, and Linxi Fan. EgoScale: Scaling dexterous manipulation with diverse egocentric human data.arX...

    2026

  56. [64]

    Graphmimic: Graph-to-graphs generative modeling from videos for policy learning

    Guangyan Chen, Te Cui, Meiling Wang, Chengcai Yang, Mengxiao Hu, Haoyang Lu, Yao Mu, Zicai Peng, Tianxing Zhou, Xinran Jiang, et al. Graphmimic: Graph-to-graphs generative modeling from videos for policy learning. In Proceedings of the Computer Vision and Pattern Recognition C...

    2025

  57. [65]

    Learning from videos through graph-to-graphs generative modeling for robotic manipulation.IEEE Transactions on Robotics, 2026

    Guangyan Chen, Meiling Wang, Te Cui, Chengcai Yang, Mengxiao Hu, Haoyang Lu, Zicai Peng, Tianxing Zhou, Xinran Jiang, Yi Yang, et al. Learning from videos through graph-to-graphs generative modeling for robotic manipulation.IEEE Transactions on Robotics, 2026

    2026

  58. [66]

    Unifying latent action and latent state pre-training for policy learning from videos

    Guangyan Chen, Meiling Wang, Te Cui, Luojie Yang, Qi Shao, Lin Zhao, Tianle Zhang, Yihang Li, Yi Yang, and Yufeng Yue. Unifying latent action and latent state pre-training for policy learning from videos. InProceedings of the SIGGRAPH Asia 2025 Conference Papers, pages 1–11, 2025

    2025

  59. [67]

    What foundation models can bring for robot learning in manipulation: A survey.The International Journal of Robotics Research, 45(7):1091–1142, 2026

    Dingzhe Li, Yixiang Jin, YuHao Sun, Yong A, Hongze Yu, Jun Shi, Xiaoshuai Hao, Peng Hao, Huaping Liu, Xiang Li, Xinde Li, Fuchun Sun, Jianwei Zhang, and Bin Fang. What foundation models can bring for robot learning in manipulation: A survey.The International Journal of Robotic...

    2026 doi

  60. [68]

    A generalist agent.Transactions on Machine Learning Research, 2022

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent.Transactions on Machine Learning Research, 2022

    2022

  61. [69]

    OpenVLA: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 25

    2024 arXiv

  62. [70]

    RDT-1B: A diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864, 2024

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-1B: A diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864, 2024

    2024 arXiv

  63. [71]

    GR00T N1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Johan Bjorck et al. GR00T N1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    2025 arXiv

  64. [72]

    DexVLA: Vision-language model with plug-in diffusion expert for general robot control

    Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. DexVLA: Vision-language model with plug-in diffusion expert for general robot control. InConference on Robot Learning, volume 305. PMLR, 2025

    2025

  65. [73]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  66. [74]

    Christensen, Hao Su, Jiajun Wu, and Yunzhu Li

    Bo Ai, Stephen Tian, Haochen Shi, Yixuan Wang, Tobias Pfaff, Cheston Tan, Henrik I. Christensen, Hao Su, Jiajun Wu, and Yunzhu Li. A review of learning-based dynamics models for robotic manipulation.Science Robotics, 10 (106):eadt1497, 2025. doi: 10.1126/scirobotics.adt1497

    2025 doi

  67. [75]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  68. [76]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. InInternational Conference on Learning Representations, 2021

    2021

  69. [77]

    Mastering diverse control tasks through world models.Nature, 640:647–653, 2025

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models.Nature, 640:647–653, 2025. arXiv:2301.04104

    2025 arXiv

  70. [78]

    Transformers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  71. [79]

    Learning latent dynamics for planning from pixels

    DanijarHafner, TimothyLillicrap, IanFischer, Ruben Villegas, David Ha, Honglak Lee, andJames Davidson. Learning latent dynamics for planning from pixels. InInternational Conference on Machine Learning, pages 2555–2565. PMLR, 2019

    2019

  72. [80]

    TD-MPC2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024

    2024

  73. [81]

    Learning universal policies via text-guided video generation

    Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. InAdvances in Neural Information Processing Systems, volume 36, pages 9156–9172, 2023

    2023

  74. [82]

    Zero-shot robotic manipulation with pre-trained image-editing diffusion models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. InInternational Conference on Learning Representations (ICLR), 2024. SuSIE; arXiv:2310.10639

    2024 arXiv

  75. [83]

    Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation.arXiv preprint arXiv:2409.16283, 2024

    Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation.arXiv preprint arXiv:2409.16283, 2024

    2024 arXiv

  76. [84]

    Dreamitate: Real-world visuomotor policy learning via video generation

    Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl Vondrick. Dreamitate: Real-world visuomotor policy learning via video generation. InProceedings of The 8th Conference on Robot Learning, volume 270 ofProceedings of Mac...

  77. [85]

    Structured world models from human videos

    Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. InProceedings of Robotics: Science and Systems (RSS), 2023. doi: 10.15607/RSS.2023.XIX.012

    2023 doi

  78. [86]

    UniSim: Learning interactive real-world simulators.arXiv preprint arXiv:2310.06114, 2023

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, and Pieter Abbeel. UniSim: Learning interactive real-world simulators.arXiv preprint arXiv:2310.06114, 2023

    2023 arXiv

  79. [87]

    Unleashing large-scale video generative pre-training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2312...

    2024 arXiv

  80. [88]

    GR-2: A generative video-language-action model with web-scale knowledge for robot manipulation.arXiv preprint arXiv:2410.06158, 2024

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, and Minzhao Zhu. GR-2: A generative video-language-action model with web-scale knowledge for robot manipulation.arXiv preprint arXiv:2410.06158, 2024

    2024 arXiv

  81. [89]

    Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

    Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

    2026 arXiv

  82. [90]

    Cosmos Policy: Fine-tuning video models for visuomotor control and planning

    Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, and Jinwei Gu. Cosmos Policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163, 2026. 26

    2026 arXiv

  83. [91]

    Wall-wm: Carving world action modeling at the event joints.arXiv preprint arXiv:2606.01955, 2026

    Shalfun Li, Victor Yao, Charles Yang, Truth Qu, Regis Cheng, Ryan Yu, Howard Lu, Newton Von, Vincent Chen, Yohann Tang, et al. Wall-wm: Carving world action modeling at the event joints.arXiv preprint arXiv:2606.01955, 2026

    2026 arXiv

  84. [92]

    Exploring the limits of vision-language-action manipulation in cross-task generalization

    Jiaming Zhou, Ke Ye, Jiayi Liu, Teli Ma, Zifan Wang, Ronghe Qiu, Kun-Yu Lin, Zhilin Zhao, and Junwei Liang. Exploring the limits of vision-language-action manipulation in cross-task generalization. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems (...

    2025

  85. [93]

    Fmimic: Foundation models are fine-grained action learners from human videos

    Guangyan Chen, Meiling Wang, Te Cui, Yao Mu, Haoyang Lu, Zicai Peng, Mengxiao Hu, Tianxing Zhou, Mengyin Fu, Yi Yang, Yufeng Yue, et al. Fmimic: Foundation models are fine-grained action learners from human videos. The International Journal of Robotics Research, page 027836492...

    2025

  86. [94]

    Vlmimic: Vision language models are visual imitation learner for fine-grained actions.Advances in Neural Information Processing Systems, 37:77860–77887, 2024

    Guangyan Chen, Meiling Wang, Te Cui, Yao Mu, Haoyang Lu, Tianxing Zhou, Zicai Peng, Mengxiao Hu, Haizhou Li, Li Yuan, et al. Vlmimic: Vision language models are visual imitation learner for fine-grained actions.Advances in Neural Information Processing Systems, 37:77860–77887, 2024

    2024

  87. [95]

    Coarse-to-fine imitation learning: Robot manipulation from a single demonstration

    Edward Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4613–4619. IEEE, 2021

    2021

  88. [96]

    OKAMI: Teaching humanoid robots manipulation skills through single video imitation

    Jinhan Li, Yifeng Zhu, Yuqi Xie, Zhenyu Jiang, Mingyo Seo, Georgios Pavlakos, and Yuke Zhu. OKAMI: Teaching humanoid robots manipulation skills through single video imitation. InConference on Robot Learning, volume 270, pages 299–317. PMLR, 2024

    2024

  89. [97]

    Learning a thousand tasks in a day.Science Robotics, 10(108):eadv7594, 2025

    Kamil Dreczkowski, Pietro Vitiello, Vitalis Vosylius, and Edward Johns. Learning a thousand tasks in a day.Science Robotics, 10(108):eadv7594, 2025. doi: 10.1126/scirobotics.adv7594

    2025 doi

  90. [98]

    Robot see robot do: Imitating articulated object manipulation with monocular 4d reconstruction

    Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, and Angjoo Kanazawa. Robot see robot do: Imitating articulated object manipulation with monocular 4d reconstruction. InConference on Robot Learning, volume 270, pages 587–603. PMLR, 2024. arXiv:2409.18121

    2024 arXiv

  91. [99]

    Zero-shot visual imitation

    Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. InInternational Conference on Learning Representations, 2018

    2018

  92. [100]

    Transformers for one-shot visual imitation

    Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. InConference on Robot Learning, pages 2071–2084. PMLR, 2021

  93. [101]

    DINOBot: Robotmanipulationviaretrievalandalignmentwithvisionfoundation models

    NormanDiPaloandEdwardJohns. DINOBot: Robotmanipulationviaretrievalandalignmentwithvisionfoundation models. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2798–2805. IEEE, 2024. arXiv:2402.13181

    2024 arXiv

  94. [102]

    See once, then act: Vision-language-action model with task learning from one-shot video demonstrations.arXiv preprint arXiv:2512.07582, 2025

    Guangyan Chen, Meiling Wang, Qi Shao, Zichen Zhou, Weixin Mao, Te Cui, Minzhao Zhu, Yinan Deng, Luojie Yang, Zhanqi Zhang, et al. See once, then act: Vision-language-action model with task learning from one-shot video demonstrations.arXiv preprint arXiv:2512.07582, 2025

    2025

  95. [103]

    Soft-DTW: A differentiable loss function for time-series

    Marco Cuturi and Mathieu Blondel. Soft-DTW: A differentiable loss function for time-series. InInternational Conference on Machine Learning, pages 894–903. PMLR, 2017

    2017

  96. [104]

    D3TW: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation

    Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3TW: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2019

  97. [105]

    Learning correspondence from the cycle-consistency of time

    Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2566–2576, 2019

    2019

  98. [106]

    GraphIRL: Graph inverse reinforcement learning from diverse videos

    Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. GraphIRL: Graph inverse reinforcement learning from diverse videos. InProceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 55–66. PM...

    2023

  99. [107]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  100. [108]

    On the continuity of rotation representations in neural networks

    Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2019. 27 Supplementary Contents

    2019

  101. [115]

    29 1.2 One-shot visual imitation

    Related work.....................................................................................29 1.1 Learning robot manipulation from demonstrations ........................................... 29 1.2 One-shot visual imitation ...................................................

  102. [116]

    Implementation details ........................................................................... 32 2.1 Alignment module...........................................................................32 2.2 Autoregressive diffusion model architecture.................................

  103. [117]

    Notation table....................................................................................38 28 Appendix A Related Work A.1 Learning Robot Manipulation from Demonstrations Learning from demonstrations offers a direct and intuitive way to teach robots new manipulation s...

  104. [2004]

    doi: 10.1016/s0896-6273(04)00181-3

    doi

  105. [2017]

    doi: 10.1109/ICRA.2017.7989384

    2017

  106. [2019]

    doi: 10.1126/scirobotics.aav3123

    doi

  107. [2021]

    doi: 10.1126/scirobotics.abd9461

    doi

  108. [2022]

    URLhttps://proceedings.mlr.press/v164/kalashnikov22a.html

  109. [2025]

    URLhttps://proceedings.mlr.press/v270/liang25b.html

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.