Pith. sign in

REVIEW 4 major objections 6 minor 65 references

GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that a latent-variable policy trained with constrained self-imitating on unlabeled demonstrations and human-intention alignment on a small labeled subset can follow text, video, and return-based instructions across four…

desk verdict A sensible weak-supervision recipe with a strong two-term ablation, but unvalidated LLM-generated Minecraft labels and sloppy reporting keep it short of acceptance. read the letter →

arxiv 2412.10410 v1 pith:OEIG65K5 submitted 2024-12-07 cs.AI cs.LGcs.RO

Shaofei Cai , Bowei Zhang , Zihao Wang , Haowei Lin , Xiaojian Ma , Anji Liu , Yitao Liang This is my paper · ORCID
classification cs.AIcs.LGcs.RO
keywords weaklysupervisedlearningmultimodalinstructionfollowinglatentvariablemodelsimitationvariationalautoencoderrobotmanipulationMinecraftbenchmarkAtari
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that instruction-following agents do not need a fully labeled dataset: a latent-variable policy trained on large amounts of unlabeled demonstrations plus a small set of text, video, or return labels can follow instructions across very different environments. The method has two parts: constrained self-imitating uses unlabeled trajectories to make the policy learn diverse behaviors while a KL term stops the latent from simply copying actions, and human intention alignment uses labeled trajectories to pull the video-conditioned latent toward the label-conditioned latent. The payoff is practical and general: experiments in Minecraft, Language Table, Simpler Env, and Atari show that GROOT-2 outperforms its self-supervised predecessor and matches stronger supervised baselines, and that adding more unlabeled data keeps helping even for tasks with no language labels. If the central claim is right, high-quality human annotations can be reserved for a small subset of trajectories, which makes large-scale instructable agents far more affordable.

What carries the argument

The central object is a shared latent intention space $Z$ with per-modality encoders $e(z\mid c)$ (video, text, return) and a latent-conditioned policy $\pi(a_t\mid o_{\le t}, z)$. Two losses shape this space. Constrained self-imitating, Eq. (2), applies to unlabeled demonstrations and adds a $\beta_1$-weighted KL penalty, $D_{\mathrm{KL}}(e(z\mid o_{1:N})\,\|\,e(z\mid o_1))$, to the behavior-cloning reconstruction term; this prevents the latent from collapsing into action-copying or ignoring observations, forcing it to carry high-level behavior information. Human intention alignment, Eq. (3), applies to labeled demonstrations and adds an MLE term, $-\beta_2\,\mathbb{E}_{z\sim \mathrm{sg}[e(z\mid w)]}[\log e(z\mid o_{1:N})]$, that pulls the video posterior toward the label posterior. During inference, any instruction is encoded as a distribution over the same $Z$, so text, reference video, and expected-return instructions can all steer the same policy.

What would settle it

Retrain GROOT-2 on the Minecraft data with the automatic labels randomly shuffled against trajectories while keeping all other losses identical; if instruction-following success on the 100-task benchmark does not clearly drop, the human-intention alignment term is not doing the claimed work.

Watch

Extended reading notes

Core claim

The paper claims that weak supervision—many unlabeled trajectories plus a few labeled ones—is enough to make a latent-variable agent truly instructable. GROOT-2's training objective is the sum of Eq. (2) and Eq. (3): on unlabeled data it learns diverse behaviors while a KL term constrains how much information the latent carries, and on labeled data the video-conditioned posterior is pushed toward the label-conditioned posterior by a closed-form MLE term. The result is a single latent space in which a text instruction, a reference video, and a desired episode return map to the same region, and the policy steered by that region generalizes to new deployments. In experiments across Minecraft, Language Table, Simpler Env, and Atari, the paper reports that GROOT-2 outperforms the self-supervised GROOT-1 on every Minecraft task family, matches or exceeds language-conditioned baselines on robot manipulation, and follows return instructions as well as video instructions on Atari.

Load-bearing premise

The load-bearing premise is that automatically generated text labels for Minecraft events genuinely capture human-level task intentions rather than only low-level per-action descriptions; if that premise is false, the alignment term is pulling the latent space toward something that is not human intention.

Editorial extensions

If this is right

  • With enough unlabeled demonstrations, instruction-following performance improves even on tasks whose training data never contained a language label.
  • A 50/50 split of labeled and unlabeled trajectories approaches the performance of fully labeled training on Language Table, so labels can be kept scarce in practice.
  • Language and video instructions share one latent space, so knowledge acquired under one modality transfers to the other.
  • On Atari, the same trained model can be steered to a requested score level by either a reference video or a numeric return, which is a step beyond winning the game.
  • Removing either training term breaks the method: without human intention alignment the video-conditioned policy mechanically copies arm motions, and without constrained self-imitating the language latent collapses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this scaling pattern holds beyond the four test environments, the main cost of building a new instructable agent shifts from annotation to collecting unlabeled behavior data, which is abundant online.
  • The closed-form MLE alignment term could be applied to other cheap label types, such as goal images or success classifiers, making the recipe even more general than the paper's text, video, and return examples.
  • The paper's latent-space spectrum suggests a practical diagnostic: monitoring the ratio of behavior-cloning to KL loss during training could tell a practitioner when the latent is degenerating into mechanical imitation or posterior collapse, though the paper does not propose such a monitoring rule.
  • A direct test of the human-intention claim would be to give the trained model human-written Minecraft instructions never seen in training and measure whether it outperforms an agent trained on the same trajectories without the alignment term; the paper does not run this exact comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces GROOT-2, a latent-variable policy trained with a weakly supervised objective. On unlabeled demonstrations, it uses constrained self-imitating (Eq. 2), a behavior-cloning loss with a KL term that regularizes the posterior over a latent intention variable. On a smaller set of labeled demonstrations (language or episode returns), it adds human intention alignment (Eq. 3): the policy is cloned conditionally on latents sampled from the label-conditional encoder, and an MLE term encourages the video-conditional posterior to assign high density to label-conditioned samples. The method is evaluated in Minecraft (an extension of the SkillForge benchmark to 100 tasks), Language Table, Simpler Env, and Atari, with text, video, and return instructions. The paper reports that GROOT-2 outperforms the self-supervised GROOT-1 and matches or exceeds prior instruction-following baselines, and that both loss terms are necessary (Table 5).

Significance. The conceptual framing is attractive: the paper identifies latent-space ambiguity in VAE-based behavior models and proposes a simple, general way to shape the latent space with sparse multi-modal labels. The breadth of evaluation (games, robotic manipulation, open-world Minecraft) strengthens the claim of generality, and the ablations in Table 5, together with the scaling experiments in Figures 8 and 9, provide useful evidence that both unlabeled and labeled data contribute. However, the central claim that Eq. (3) aligns the latent space with human intention is only as strong as the labels used in Dlab. For Minecraft, the largest and most open-ended evaluation, the labels are chatGPT-generated from event metadata (Appendix C) and are not validated; this is a load-bearing gap. The statistical support for a few headline comparisons (Table 2) is also weaker than the text implies. These issues are addressable and do not undermine the overall approach, but they must be resolved before the contribution can be fully accepted.

major comments (4)
  1. [Section 3 (Eq. 3) and Appendix C] Appendix C states that Minecraft text labels are produced by using chatGPT to expand event metadata (craft, pickup, mine block, drop, kill, use, custom), with 128-frame segments constructed around each event. No human validation, noise analysis, or comparison with alternative labels is provided. Because Eq. (3) maximizes the likelihood of the video posterior e(z|o1:N) at samples from the label-conditioned distribution e(z|w1:M), any systematic label bias—e.g., event-level descriptions such as 'mine block: cobblestone' instead of task-level intentions such as 'collect wood to craft a tool'—is propagated into the shared latent space. This directly affects the interpretation of the Minecraft results in Table 1, which is the largest and most open-ended evaluation in the paper. Please add a human validation study of the chatGPT labels, an analysis of their task-level vs. event-level content, or an ablation with a small set of manually verified labels.
  2. [Section 4, 'Results on the Language Table benchmark' and Table 2] The text claims that GROOT-2 'leads by an absolute success rate of 4% following text-based instructions,' but the reported uncertainties do not support this: the overall text-conditioned success rates are 84±6 for GROOT-2 (100%) and 82±8 for GROOT-2 (50%), versus 78±4 for LAVA, and several task families have substantially overlapping error bars. Please provide paired statistics, per-task confidence intervals, or a more cautious wording of the comparison.
  3. [Section 4, 'Results on the Open-World Minecraft Benchmark' and Appendix C] The extension of the Minecraft SkillForge benchmark from 30 to 100 tasks is not described in sufficient detail to be reproducible. Appendix C covers only the training dataset; there is no task list, no success criterion for each of the eight task families, and no evaluation protocol beyond '30 times across three seeds.' Please add a complete task specification and, ideally, the evaluation code, or clearly cite a public benchmark definition that contains these details.
  4. [Section 3 (Eq. 3) and Table 5] The ablation of the alignment loss removes the entire Llab term, which includes both the label-conditioned behavior-cloning term and the MLE alignment term. Table 5 therefore does not isolate the contribution of the MLE term, which is the novel mechanism claimed to align the latent space with human intentions. Please add an ablation that keeps the label-conditioned behavior-cloning objective but removes only the MLE term (or vice versa) to verify that the alignment term itself, rather than the additional labeled imitation data, drives the reported gains.
minor comments (6)
  1. [Table 2 and surrounding text] The text says 'We mark the results of RT-2 in gray here,' but the table lists RT-1 and no gray shading is visible; please reconcile the model name and formatting.
  2. [Section 4, 'How Does Language and Video Losses Impact Performance?'] The sentence 'the variant without Llab loss failed to complete any tasks' is contradicted by Table 5, which reports 10±2% (vision) and 12±3% (lang) success for that variant; please correct the wording.
  3. [Section 4 and Appendix B] The main-text Atari experiment is described as using four games (Breakout, Demon Attack, Hero, Name This Game), while Appendix B reports joint training on 35 games; please clarify whether the four-game result is a subset of the 35-game experiment and state the relationship explicitly.
  4. [Table 5] The layout of Table 5, with 'Variants −Llab baseline −Ldem baseline' in a single row, is confusing; please separate the two ablations into distinct rows with explicit full-model columns.
  5. [References] The references 'Lynch et al., 2020a,' 'Lynch et al., 2020b,' and 'Lynch et al., 2020c' all point to the same paper (Learning latent plans from play); please consolidate into a single citation to avoid implying three distinct works.
  6. [Appendix C] The sentence 'across all 1518 events, 414,387 segments are included' is ambiguous, because a cap of 2000 segments per event would allow up to roughly 3 million segments; please clarify whether 1518 is the number of event types or the number of events with nonempty label sets, and explain why the realized total is much lower.

Circularity Check

1 steps flagged · score 2.0 of 10

Central benchmark claims are external and not circular; one ancillary latent-visualization result is enforced by the Eq. (3) training objective.

  1. fitted input called prediction [Section 3, Eq. (3); Section 4, 'What Does the Visualization of the Learned Latent Space Reveal?' and Figure 7.]
    "The MLE-based alignment objective ensures that the latent sampled from the label-conditioned distribution e(z|w1:M) can also be sampled from its video-conditioned distribution e(z|o1:N). ... The second row shows results for GROOT (wsl), which captures continuous variations in video behavior quality across all games."

    Eq. (3) contains the term -beta2 * E_{z ~ sg[e(z|w1:M)]}[log e(z|o1:N)], which directly trains the video posterior to put high density on latents sampled from the label-conditioned distribution. In the Atari experiments the label w is the normalized episode return, so the t-SNE result that video embeddings align with reward labels is a direct visualization of this fitted objective rather than an emergent validation. The same is true for the statement that 'the video encoder and reward encoder share the same intention space.' This is a fitted-input-called-prediction step, but it is ancillary: the headline results in Tables 1-3 are external success-rate benchmarks and do not reduce to the loss.

full rationale

I find no significant circularity in the central derivation. The weakly supervised objective in Eqs. (2)-(4) is a beta-VAE behavior-cloning objective plus a cross-modal MLE alignment term, and the headline claims are tested on external benchmarks: Minecraft SkillForge task families, Language Table, Simpler Env, and Atari return/video conditioning. Those success rates are measured in held-out environments and are not derivable from the loss alone. The comparisons with GROOT-1 and VPT are legitimate external baselines, and the self-citations to Cai et al. (2023b) are used to define the GROOT-1 baseline and the Minecraft benchmark rather than as an unverified uniqueness theorem. The unvalidated chatGPT-generated Minecraft labels in Appendix C are a data-quality and validity risk for the 'human intention' interpretation, but not a circularity, because the labels are external inputs generated from event metadata rather than from the model's outputs. The one mild by-construction element is the t-SNE visualization in Figure 7: because Eq. (3) directly fits the video posterior toward label-conditioned latents, observing reward-aligned clusters is expected from training. This element is ancillary; reframing or removing it would not undermine the main empirical claims, which rest on external task success rates.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method builds on standard VAE machinery plus two hand-set weights and a specific, non-derived alignment loss. The most fragile input is the automatically generated text labels, which carry the entire human-intention signal.

free parameters (2)
  • beta1 = 0.1
    Weight on the KL term in Eq. 2 (constrained self-imitating); set by hand without sensitivity analysis.
  • beta2 = 0.1
    Weight on the MLE alignment term in Eq. 3 (human intention alignment); set by hand without sensitivity analysis.
assumptions (4)
  • standard math The ELBO is a valid training objective for latent-conditioned trajectory models
    Used in Eq. 1 and Eq. 2; foundation of the VAE approach.
  • domain assumption The inverse dynamics model of each environment is easy to learn, so the posterior encoder can be conditioned on observations alone
    Section 2.1 states this assumption and excludes actions from the posterior input.
  • domain assumption chatGPT-generated labels from event metadata are adequate proxies for human intention
    Appendix C describes label generation; quality is not validated.
  • ad hoc to paper The MLE term with stop-gradient (Eq. 3) is an effective alignment mechanism for human intention
    Section 3 introduces this objective without derivation or comparison to alternative alignment losses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents." pith.science (2026). https://pith.science/paper/OEIG65K5

@misc{pith2026241210410,
  author       = {Pith},
  title        = {Pith review of: GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OEIG65K5}},
  note         = {Machine review of arXiv:2412.10410}
}
read the original abstract

Developing agents that can follow multimodal instructions remains a fundamental challenge in robotics and AI. Although large-scale pre-training on unlabeled datasets (no language instruction) has enabled agents to learn diverse behaviors, these agents often struggle with following instructions. While augmenting the dataset with instruction labels can mitigate this issue, acquiring such high-quality annotations at scale is impractical. To address this issue, we frame the problem as a semi-supervised learning task and introduce GROOT-2, a multimodal instructable agent trained using a novel approach that combines weak supervision with latent variable models. Our method consists of two key components: constrained self-imitating, which utilizes large amounts of unlabeled demonstrations to enable the policy to learn diverse behaviors, and human intention alignment, which uses a smaller set of labeled demonstrations to ensure the latent space reflects human intentions. GROOT-2's effectiveness is validated across four diverse environments, ranging from video games to robotic manipulation, demonstrating its robust multimodal instruction-following capabilities.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 19 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Multi-objective latent space optimization of generative molecular design models

    ANM Nafiz Abeer, Nathan M Urban, M Ryan Weil, Francis J Alexander, and Byung-Jun Yoon. Multi-objective latent space optimization of generative molecular design models. Patterns, 2024

    work page 2024

  3. [3]

    Imitating interactive intelligence

    Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020

    arXiv 2012

  4. [4]

    An optimistic perspective on offline reinforcement learning

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp.\ 104--114. PMLR, 2020

    2020

  5. [5]

    Opal: Offline primitive discovery for accelerating offline reinforcement learning

    Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020

    arXiv 2010

  6. [6]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, R...

    arXiv 2022

  7. [7]

    Fixing a broken elbo

    Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. In International conference on machine learning, pp.\ 159--168. PMLR, 2018

    2018

  8. [8]

    Abbeel, and Wojciech Zaremba

    Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojciech Zaremba. Hindsight experience replay. ArXiv, abs/1707.01495, 2017. URL https://api.semanticscholar.org/CorpusID:3532908

    arXiv 2017

Show all 65 references
  1. [9]

    Agent57: Outperforming the atari human benchmark

    Adri \`a Puigdom \`e nech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International conference on machine learning, pp.\ 507--517. PMLR, 2020

    2020

  2. [10]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. ArXiv, abs/2206.11795, 2022. URL https://api.semanticscholar.org/Co...

    2022 arXiv

  3. [11]

    The arcade learning environment: An evaluation platform for general agents

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253--279, 2013

    2013

  4. [12]

    Joshi, Ryan C

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kal...

    2022 arXiv

  5. [13]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  6. [14]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jef...

    2005 arXiv

  7. [15]

    Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction

    Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 13734--13744, 2023 a . UR...

    2023

  8. [16]

    Groot: Learning to follow instructions by watching gameplay videos

    Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Groot: Learning to follow instructions by watching gameplay videos. In The Twelfth International Conference on Learning Representations, 2023 b

    2023

  9. [17]

    Abbeel, A

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org...

    2021

  10. [19]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Jan 2019. doi:10.18653...

    2019 doi

  11. [20]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019. URL https://api.semanticscholar.org/CorpusID:52967399

    2019 arXiv

  12. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2010 arXiv

  13. [22]

    One-shot imitation learning

    Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017

    2017

  14. [23]

    Implicit deep latent variable models for text generation

    Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. Implicit deep latent variable models for text generation. arXiv preprint arXiv:1908.11527, 2019

    1908 arXiv

  15. [24]

    Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning

    Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019

    1910 arXiv

  16. [25]

    Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela M

    William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela M. Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. In International Joint Conference on Artificial Intelligence, 2019. URL https://api.semanticsc...

    2019

  17. [26]

    Vln-bert: A recurrent vision-and-language bert for navigation

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. Vln-bert: A recurrent vision-and-language bert for navigation. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 1643--1653, 2020. URL https://api.semanticscholar.org/Co...

    2021

  18. [27]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023

    2023 arXiv

  19. [28]

    Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers

    Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, et al. Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers. arXiv preprint arXiv:2403.12943, 2024

    2024 arXiv

  20. [29]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. ArXiv, abs/2202.02005, 2022. URL https://api.semanticscholar.org/CorpusID:237257594

    2022 arXiv

  21. [30]

    The malmo platform for artificial intelligence experimentation

    Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In International Joint Conference on Artificial Intelligence, 2016. URL https://api.semanticscholar.org/CorpusID:9953039

    2016

  22. [31]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. URL https://api.semanticscholar.org/CorpusID:216078090

    2013 arXiv

  23. [32]

    Multi-game decision transformers

    Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35: 0 27921--27936, 2022

    2022

  24. [33]

    Evaluating real-world robot manipulation policies in simulation

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv...

    2024 arXiv

  25. [34]

    McIlraith

    Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila A. McIlraith. Steve-1: A generative model for text-to-behavior in minecraft. ArXiv, abs/2306.00937, 2023. URL https://api.semanticscholar.org/CorpusID:258999563

    2023 arXiv

  26. [36]

    Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023

    2023 arXiv

  27. [37]

    Language conditioned imitation learning over unstructured data

    Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648, 2020

    2005 arXiv

  28. [38]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp.\ 1113--1132. PMLR, 2020 a

    2020

  29. [39]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp.\ 1113--1132. PMLR, 2020 b

    2020

  30. [40]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp.\ 1113--1132. PMLR, 2020 c

    2020

  31. [41]

    Interactive language: Talking to robots in real time

    Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023

    2023

  32. [42]

    Zson: Zero-shot object-goal navigation using multimodal goal embeddings

    Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. ArXiv, abs/2206.12403, 2022. URL https://api.semanticscholar.org/CorpusID:250048645

    2022 arXiv

  33. [43]

    What matters in language conditioned robotic imitation learning over unstructured data

    Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7 0 (4): 0 11205--11212, 2022

    2022

  34. [44]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wi...

    2015

  35. [45]

    Goal representations for instruction following: A semi-supervised language interface to control

    Vivek Myers, Andre Wang He, Kuan Fang, Homer Rich Walke, Philippe Hansen-Estruch, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. Goal representations for instruction following: A semi-supervised language interface to control. In Conference on ...

    2023

  36. [46]

    Octo: An open-source generalist robot policy

    Octo Model Team , Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Oct...

    2024

  37. [47]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023

    arXiv

  38. [48]

    Conditional variational autoencoder for neural machine translation

    Artidoro Pagnoni, Kevin Liu, and Shangyan Li. Conditional variational autoencoder for neural machine translation. arXiv preprint arXiv:1812.04405, 2018

    2018 arXiv

  39. [49]

    Episodic transformer for vision-and-language navigation

    Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 15942--15952, 2021

    2021

  40. [50]

    Accelerating reinforcement learning with learned skill priors

    Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp.\ 188--204. PMLR, 2021

    2021

  41. [51]

    Scaling instructable agents across many simulated worlds

    Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404.10179, 2024

    2024 arXiv

  42. [52]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  43. [53]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  44. [54]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  45. [55]

    A generalist agent

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022

    2022 arXiv

  46. [56]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, ...

    2019

  47. [57]

    Reinforcement learning upside down: Don't predict rewards - just map them to actions

    Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards - just map them to actions. ArXiv, abs/1912.02875, 2019. URL https://api.semanticscholar.org/CorpusID:208857600

    1912 arXiv

  48. [58]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  49. [59]

    Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Shawn Ma, and Yitao Liang. Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents. Advances in Neural Information Processing Systems, 36, 2023 a

    2023

  50. [60]

    Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models

    Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997, 2023 b

    2023 arXiv

  51. [61]

    Xskill: Cross embodiment skill discovery

    Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pp.\ 3536--3555. PMLR, 2023

    2023

  52. [62]

    Julian, Karol Hausman, Chelsea Finn, and Sergey Levine

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan C. Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. ArXiv, abs/1910.10897, 2019. URL https://api.semanticscholar.org/CorpusID:204852201

    1910 arXiv

  53. [63]

    Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining

    Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In European Conference on Computer Vision, 2022. URL https://api.semanticscholar.org/CorpusID:250626771

    2022

  54. [64]

    Hierarchical task learning from language instructions with unified transformers and self-monitoring

    Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monitoring. arXiv preprint arXiv:2106.03427, 2021

    2021 arXiv

  55. [65]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  56. [66]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  57. [67]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2021 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.