{"id":"c0bd119a-1b6e-442d-ac31-26b06088df42","arxiv_id":"2412.10410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A weakly supervised latent-variable agent improves multimodal instruction following by combining VAE self-imitating on unlabeled data with a likelihood-based alignment of labeled and video latents.","lead":"GROOT-2 trains game and robot agents to follow text, video, and score instructions using mostly unlabeled demonstrations plus a small set of labeled ones. It aligns video and label encodings into one intention space, improving instruction following across Minecraft, Atari, and tabletop robot benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Minecraft chatGPT-generated text labels are unvalidated and may encode event-level actions rather than task-level human intentions, undermining the alignment claim behind Eq. 3.","rationale":"The reader's weakest assumption matches the most load-bearing concern I can identify: the unvalidated chatGPT labels in Appendix C are the only source of 'human intention' supervision for the Minecraft experiments, and Eq. 3 explicitly propagates whatever bias those labels contain into the shared latent space. This is not a mere reporting problem; it threatens the conceptual claim that weak supervision aligns the latent space with human intentions. At the same time, the concern is localized: Language Table uses oracle-generated labels, Simpler Env uses OpenX labels, and Atari uses episode returns, so the broader claim of multimodal instruction following does not collapse entirely. The reader's conditional verdict remains appropriate—the paper's central claim is plausible but not fully established until label quality is validated. I would not escalate to rejection because the paper's ablations (Table 5), scaling results, and cross-environment demonstrations provide independent support for the method's basic mechanics, and the label concern is addressable with a concrete audit and retraining comparison. The main secondary issues—missing RT-2 results, the 4-versus-35 Atari discrepancy, and unreleased code—reinforce the need for conditions but do not change the verdict.","tokens_in":21926,"tokens_out":4978,"duration_ms":53249,"concrete_test":"Sample 200 of the 414,387 chatGPT-labeled Minecraft segments described in Appendix C and have three independent human annotators classify each label as (a) a task-level intention, (b) an event/action-level description, or (c) invalid or inconsistent with the video. If a majority are (b) or (c), retrain GROOT-2 on a filtered set containing only task-level labels and rerun the Table 1 Minecraft benchmark. If performance on text- and video-conditioned tasks does not drop materially, the label-quality concern does not land; if performance drops, the alignment claim is not established for Minecraft.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Eq. 3 aligns GROOT-2's latent space with human intention depends on the quality of the labels in Dlab. For the largest and most open-ended evaluation (Minecraft, Table 1), those labels are not human annotations: Appendix C states that chatGPT was used to expand event metadata (craft, pickup, mine block, drop, kill, use, custom), producing 414,387 overlapping 128-frame segments. No human validation, noise analysis, or comparison against non-chatGPT labels is provided. Because the MLE alignment term in Eq. 3 forces the video posterior e(z|o1:N) to assign high density to samples from e(z|w1:M), any systematic bias in w—for example, labels that describe the immediately mined block rather than the task-level goal like 'collect wood to craft a tool'—is propagated into the shared latent space. If the Minecraft labels are event-level, then GROOT-2's text-conditioned and visual-conditioned gains in Table 1 may partly reflect imitation of event sketches rather than human intentions, which is precisely the failure mode the paper claims to fix. The claim is therefore conditionally supported: it holds only if chatGPT labels encode task-level intentions, and the paper currently supplies no evidence for that.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GROOT-2, a latent-variable policy trained with a weakly supervised objective. On unlabeled demonstrations, it uses constrained self-imitating (Eq. 2), a behavior-cloning loss with a KL term that regularizes the posterior over a latent intention variable. On a smaller set of labeled demonstrations (language or episode returns), it adds human intention alignment (Eq. 3): the policy is cloned conditionally on latents sampled from the label-conditional encoder, and an MLE term encourages the video-conditional posterior to assign high density to label-conditioned samples. The method is evaluated in Minecraft (an extension of the SkillForge benchmark to 100 tasks), Language Table, Simpler Env, and Atari, with text, video, and return instructions. The paper reports that GROOT-2 outperforms the self-supervised GROOT-1 and matches or exceeds prior instruction-following baselines, and that both loss terms are necessary (Table 5).","tokens_in":21985,"tokens_out":10125,"duration_ms":87778,"significance":"The conceptual framing is attractive: the paper identifies latent-space ambiguity in VAE-based behavior models and proposes a simple, general way to shape the latent space with sparse multi-modal labels. The breadth of evaluation (games, robotic manipulation, open-world Minecraft) strengthens the claim of generality, and the ablations in Table 5, together with the scaling experiments in Figures 8 and 9, provide useful evidence that both unlabeled and labeled data contribute. However, the central claim that Eq. (3) aligns the latent space with human intention is only as strong as the labels used in Dlab. For Minecraft, the largest and most open-ended evaluation, the labels are chatGPT-generated from event metadata (Appendix C) and are not validated; this is a load-bearing gap. The statistical support for a few headline comparisons (Table 2) is also weaker than the text implies. These issues are addressable and do not undermine the overall approach, but they must be resolved before the contribution can be fully accepted.","major_comments":[{"comment":"Appendix C states that Minecraft text labels are produced by using chatGPT to expand event metadata (craft, pickup, mine block, drop, kill, use, custom), with 128-frame segments constructed around each event. No human validation, noise analysis, or comparison with alternative labels is provided. Because Eq. (3) maximizes the likelihood of the video posterior e(z|o1:N) at samples from the label-conditioned distribution e(z|w1:M), any systematic label bias—e.g., event-level descriptions such as 'mine block: cobblestone' instead of task-level intentions such as 'collect wood to craft a tool'—is propagated into the shared latent space. This directly affects the interpretation of the Minecraft results in Table 1, which is the largest and most open-ended evaluation in the paper. Please add a human validation study of the chatGPT labels, an analysis of their task-level vs. event-level content, or an ablation with a small set of manually verified labels.","section":"Section 3 (Eq. 3) and Appendix C"},{"comment":"The text claims that GROOT-2 'leads by an absolute success rate of 4% following text-based instructions,' but the reported uncertainties do not support this: the overall text-conditioned success rates are 84±6 for GROOT-2 (100%) and 82±8 for GROOT-2 (50%), versus 78±4 for LAVA, and several task families have substantially overlapping error bars. Please provide paired statistics, per-task confidence intervals, or a more cautious wording of the comparison.","section":"Section 4, 'Results on the Language Table benchmark' and Table 2"},{"comment":"The extension of the Minecraft SkillForge benchmark from 30 to 100 tasks is not described in sufficient detail to be reproducible. Appendix C covers only the training dataset; there is no task list, no success criterion for each of the eight task families, and no evaluation protocol beyond '30 times across three seeds.' Please add a complete task specification and, ideally, the evaluation code, or clearly cite a public benchmark definition that contains these details.","section":"Section 4, 'Results on the Open-World Minecraft Benchmark' and Appendix C"},{"comment":"The ablation of the alignment loss removes the entire Llab term, which includes both the label-conditioned behavior-cloning term and the MLE alignment term. Table 5 therefore does not isolate the contribution of the MLE term, which is the novel mechanism claimed to align the latent space with human intentions. Please add an ablation that keeps the label-conditioned behavior-cloning objective but removes only the MLE term (or vice versa) to verify that the alignment term itself, rather than the additional labeled imitation data, drives the reported gains.","section":"Section 3 (Eq. 3) and Table 5"}],"minor_comments":[{"comment":"The text says 'We mark the results of RT-2 in gray here,' but the table lists RT-1 and no gray shading is visible; please reconcile the model name and formatting.","section":"Table 2 and surrounding text"},{"comment":"The sentence 'the variant without Llab loss failed to complete any tasks' is contradicted by Table 5, which reports 10±2% (vision) and 12±3% (lang) success for that variant; please correct the wording.","section":"Section 4, 'How Does Language and Video Losses Impact Performance?'"},{"comment":"The main-text Atari experiment is described as using four games (Breakout, Demon Attack, Hero, Name This Game), while Appendix B reports joint training on 35 games; please clarify whether the four-game result is a subset of the 35-game experiment and state the relationship explicitly.","section":"Section 4 and Appendix B"},{"comment":"The layout of Table 5, with 'Variants −Llab baseline −Ldem baseline' in a single row, is confusing; please separate the two ablations into distinct rows with explicit full-model columns.","section":"Table 5"},{"comment":"The references 'Lynch et al., 2020a,' 'Lynch et al., 2020b,' and 'Lynch et al., 2020c' all point to the same paper (Learning latent plans from play); please consolidate into a single citation to avoid implying three distinct works.","section":"References"},{"comment":"The sentence 'across all 1518 events, 414,387 segments are included' is ambiguous, because a cap of 2000 segments per event would allow up to roughly 3 million segments; please clarify whether 1518 is the number of event types or the number of events with nonempty label sets, and explain why the realized total is much lower.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the proposed method is interesting. The main concern to weigh is that the headline claim of 'human intention alignment' rests on unvalidated chatGPT labels for the Minecraft experiments; the authors should be asked for a validation study before acceptance. The under-specified benchmark extension and the overlapping confidence intervals in Table 2 are secondary but should also be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable extension of latent-plan VAE work to weakly supervised instruction following. The two-term objective—constrained self-imitating on unlabeled demos and human intention alignment on labeled ones—is clean, and Table 5 does real work: removing either term collapses the corresponding modality. That alone justifies reading the paper.\n\nWhat's actually new: framing label-space alignment as an MLE term between label- and video-conditioned latents, with a stop-gradient on the label side. It's not a huge theoretical step beyond semi-supervised VAEs (Kingma et al. 2014 should be cited), but it's a sensible configuration and the integration with a Transformer-XL policy is nontrivial. The broad evaluation across Minecraft, Language Table, Simpler Env, and Atari is a plus, and the scaling curves are directionally consistent with the weak-supervision story.\n\nSoft spots, in decreasing seriousness. First, the Minecraft text labels: Appendix C says chatGPT was used to expand event metadata (craft, pickup, mine block, etc.), producing segments centered on events. Those are event-level descriptions, not necessarily task-level intentions. Eq. 3 forces the video latent to match them, and the paper gives no human validation, no noise analysis, no comparison against alternative labeling. If the labels are \"mine cobblestone\" rather than \"build a tunnel\", the alignment signal is biased toward action sketches—which is exactly the failure mode the paper claims to fix. This is the load-bearing weakness. The claim could still hold, but the current evidence doesn't establish it. Second, several reporting inconsistencies: the RT-2 baseline is mentioned but no numbers appear in Table 2; main text Atari says four games but Appendix B trains and reports on 35; the \"100-task Minecraft benchmark\" is referenced but the task list isn't in Appendix C. Third, no code or weights. For an empirical paper of this scope, that hurts reproducibility.\n\nThe error-bar overlaps with LAVA and RT-1-X on Language Table are real but not damning—the method is in the same ballpark with half the labels, which is the interesting claim. The visualization section is partly circular because the MLE loss directly encourages latent closeness, but it's presented as illustration, not evidence.\n\nBottom line: the core recipe is plausible and the ablations are informative. The label-quality gap needs direct evidence, and the reporting needs cleanup. I'd send this to review; a careful referee could push it to acceptance with revisions. I'd bring it to reading group for the weak-supervision discussion, and I'd cite the method in related work.","headline":"A sensible weak-supervision recipe with a strong two-term ablation, but unvalidated LLM-generated Minecraft labels and sloppy reporting keep it short of acceptance.","tokens_in":22686,"tokens_out":3190,"would_cite":true,"duration_ms":28530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a latent-variable policy trained with constrained self-imitating on unlabeled demonstrations and human-intention alignment on a small labeled subset can follow text, video, and return-based instructions across four…","keywords":["weakly supervised learning","multimodal instruction following","latent variable models","imitation learning","variational autoencoder","robot manipulation","Minecraft benchmark","Atari"],"falsifier":"Retrain GROOT-2 on the Minecraft data with the automatic labels randomly shuffled against trajectories while keeping all other losses identical; if instruction-following success on the 100-task benchmark does not clearly drop, the human-intention alignment term is not doing the claimed work.","tokens_in":21553,"feed_emoji":"🤖","tokens_out":9116,"duration_ms":83722,"temperature":0.7,"pith_summary":"This paper tries to establish that instruction-following agents do not need a fully labeled dataset: a latent-variable policy trained on large amounts of unlabeled demonstrations plus a small set of text, video, or return labels can follow instructions across very different environments. The method has two parts: constrained self-imitating uses unlabeled trajectories to make the policy learn diverse behaviors while a KL term stops the latent from simply copying actions, and human intention alignment uses labeled trajectories to pull the video-conditioned latent toward the label-conditioned latent. The payoff is practical and general: experiments in Minecraft, Language Table, Simpler Env, and Atari show that GROOT-2 outperforms its self-supervised predecessor and matches stronger supervised baselines, and that adding more unlabeled data keeps helping even for tasks with no language labels. If the central claim is right, high-quality human annotations can be reserved for a small subset of trajectories, which makes large-scale instructable agents far more affordable.","feed_headline":"Light labeling makes one agent follow text, video, and score goals","feed_subtitle":"On Minecraft, robotics, and Atari benchmarks, unlabeled demos plus a small labeled set suffice.","key_machinery":"The central object is a shared latent intention space $Z$ with per-modality encoders $e(z\\mid c)$ (video, text, return) and a latent-conditioned policy $\\pi(a_t\\mid o_{\\le t}, z)$. Two losses shape this space. Constrained self-imitating, Eq. (2), applies to unlabeled demonstrations and adds a $\\beta_1$-weighted KL penalty, $D_{\\mathrm{KL}}(e(z\\mid o_{1:N})\\,\\|\\,e(z\\mid o_1))$, to the behavior-cloning reconstruction term; this prevents the latent from collapsing into action-copying or ignoring observations, forcing it to carry high-level behavior information. Human intention alignment, Eq. (3), applies to labeled demonstrations and adds an MLE term, $-\\beta_2\\,\\mathbb{E}_{z\\sim \\mathrm{sg}[e(z\\mid w)]}[\\log e(z\\mid o_{1:N})]$, that pulls the video posterior toward the label posterior. During inference, any instruction is encoded as a distribution over the same $Z$, so text, reference video, and expected-return instructions can all steer the same policy.","core_discovery":"The paper claims that weak supervision—many unlabeled trajectories plus a few labeled ones—is enough to make a latent-variable agent truly instructable. GROOT-2's training objective is the sum of Eq. (2) and Eq. (3): on unlabeled data it learns diverse behaviors while a KL term constrains how much information the latent carries, and on labeled data the video-conditioned posterior is pushed toward the label-conditioned posterior by a closed-form MLE term. The result is a single latent space in which a text instruction, a reference video, and a desired episode return map to the same region, and the policy steered by that region generalizes to new deployments. In experiments across Minecraft, Language Table, Simpler Env, and Atari, the paper reports that GROOT-2 outperforms the self-supervised GROOT-1 on every Minecraft task family, matches or exceeds language-conditioned baselines on robot manipulation, and follows return instructions as well as video instructions on Atari.","pith_inferences":["If this scaling pattern holds beyond the four test environments, the main cost of building a new instructable agent shifts from annotation to collecting unlabeled behavior data, which is abundant online.","The closed-form MLE alignment term could be applied to other cheap label types, such as goal images or success classifiers, making the recipe even more general than the paper's text, video, and return examples.","The paper's latent-space spectrum suggests a practical diagnostic: monitoring the ratio of behavior-cloning to KL loss during training could tell a practitioner when the latent is degenerating into mechanical imitation or posterior collapse, though the paper does not propose such a monitoring rule.","A direct test of the human-intention claim would be to give the trained model human-written Minecraft instructions never seen in training and measure whether it outperforms an agent trained on the same trajectories without the alignment term; the paper does not run this exact comparison."],"forward_implications":["With enough unlabeled demonstrations, instruction-following performance improves even on tasks whose training data never contained a language label.","A 50/50 split of labeled and unlabeled trajectories approaches the performance of fully labeled training on Language Table, so labels can be kept scarce in practice.","Language and video instructions share one latent space, so knowledge acquired under one modality transfers to the other.","On Atari, the same trained model can be steered to a requested score level by either a reference video or a numeric return, which is a step beyond winning the game.","Removing either training term breaks the method: without human intention alignment the video-conditioned policy mechanically copies arm motions, and without constrained self-imitating the language latent collapses."],"supporting_citations":[{"why":"Supplies the large unlabeled Minecraft contractor dataset and the VPT baseline, the main testbed for open-world instruction following.","marker":"Baker et al. (2022)"},{"why":"Defines GROOT-1, the self-supervised video-conditioned predecessor that GROOT-2 must beat, and the original Minecraft SkillForge benchmark that the paper extends to 100 tasks.","marker":"Cai et al. (2023b)"},{"why":"Provides the Language Table dataset, environment, and LAVA baseline used to test language and video instruction following in tabletop manipulation.","marker":"Lynch et al. (2023)"},{"why":"Provides STEVE-1, the text- and image-conditioned Minecraft baseline whose success rates GROOT-2 compares against.","marker":"Lifshitz et al. (2023)"},{"why":"Provides RT-1, the language-conditioned transformer policy used as a robotic-manipulation baseline on Language Table and Simpler Env.","marker":"Brohan et al. (2022)"},{"why":"Supplies the offline Atari replay-buffer trajectories with episode returns used to train and evaluate return- and video-conditioned steering.","marker":"Agarwal et al. (2020)"},{"why":"Provides the Simpler Env simulator and benchmark used to evaluate GROOT-2 on simulated robot arm tasks.","marker":"Li et al. (2024)"},{"why":"Gives the variational autoencoder framework, ELBO, and reparameterization trick underlying both training objectives.","marker":"Kingma & Welling (2013)"}],"fun_headline_variants":["Few labels, many demos: one agent for text, video, and return goals","GROOT-2: weak supervision enough for multimodal instruction following","Many unlabeled demos, few labels: how to make instructable robots","One latent space unifies text, video, and return goals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that automatically generated text labels for Minecraft events genuinely capture human-level task intentions rather than only low-level per-action descriptions; if that premise is false, the alignment term is pulling the latent space toward something that is not human intention.","fun_headline_variants_meta":{"raw":{"variants":["Few labels, many demos: one agent for text, video, and return goals","GROOT-2: weak supervision enough for multimodal instruction following","Many unlabeled demos, few labels: how to make instructable robots","One latent space unifies text, video, and return goals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000668,"raw_usage":{"total_tokens":3031,"prompt_tokens":914,"completion_tokens":2117,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2050}},"tokens_in":530,"tokens_out":2117,"duration_ms":16065,"temperature":1.0,"reasoning_tokens":2050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:36:28.429679+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain GROOT-2 on the Minecraft data with the automatic labels randomly shuffled against trajectories while keeping all other losses identical; if instruction-following success on the 100-task benchmark does not clearly drop, the human-intention alignment term is not doing the claimed work.","supporting_citations":[],"review_version":1}