{"id":"9255b747-4956-4959-9504-43077b8a34ff","arxiv_id":"2412.00949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An audio-conditioned STEVE-1 agent, built with a new Minecraft audio-video CLIP model and a learned prior, matches or beats text- and video-conditioned versions on most short-horizon collection tasks.","lead":"The authors extended the Minecraft agent STEVE-1 so it can follow audio instructions, not just text and images. The work shows sound can be a cheap, natural way to command embodied agents, though performance depends on task-specific training data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The audio-conditioned performance gain in §4.1 is not separable from task-specific training data: Appendix A discloses a 2-hour augmentation relevant to the evaluated tasks and a YouTube-only failure that lacks the cited results section.","rationale":"The reader's weakest assumption pointed to the same evidence: alignment of Audio-Video CLIP to MineCLIP is not demonstrated on YouTube data alone, and task-specific data was added after the YouTube-only version underperformed. I agree and would frame it as a training/evaluation overlap confound that undermines the strength of the §4.1 claim, but not the methodology's overall plausibility. The paper deserves credit for disclosing the augmentation in Appendix A and for specifying the model architectures, but the referenced YouTube-only control is missing and no significance testing is reported for the 10-seed evaluations. The promised open-source code and checkpoints make the proposed ablation runnable. Because the verdict is already CONDITIONAL, my read does not move it; it reinforces the condition that the authors should release the control experiment and clarify the training/evaluation separation.","tokens_in":9119,"tokens_out":5602,"duration_ms":55324,"concrete_test":"Run the Figure 4 evaluation protocol with the Audio-Video CLIP checkpoint trained on the 25 hours of YouTube data only, i.e., before adding the two hours of task-specific videos described in Appendix A, and compare the six per-task collection counts against the full-data checkpoint and the text/visual baselines. If the YouTube-only audio-conditioned agent no longer matches or exceeds the text/visual baselines, the central claim depends on the task-specific augmentation rather than the general audio-to-MineCLIP mapping.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1's load-bearing claim — that the audio-conditioned STEVE-1 agent 'generally performs better than the original STEVE-1 modalities' — rests on an uneven comparison. The Audio-Video CLIP model was trained on a dataset that includes two hours of task-specific gameplay 'more specifically relevant to our evaluated set of tasks' (Appendix A). The six evaluation tasks are exactly the item-collection behaviors (dirt, wood, seeds, sand, cobblestone, leaves) that such videos would depict, so the audio-conditioned advantage may reflect that the audio and video encoders were exposed to the evaluation-task distribution during CLIP pretraining. The text and visual STEVE-1 baselines received no such task-specific exposure, making the comparison uneven. This concern is sharpened by the paper's own appendix: 'we found that using YouTube data exclusively yielded an underperforming audio-conditioned STEVE-1 agent (see Experimental results section below)' — and yet no such 'Experimental results' section appears in the paper. Thus the key control demonstrating that the method fails without task-specific data is referenced but not presented. The prior network's training objective (loss, paired data, and whether it is supervised with MineCLIP video embeddings) is also not specified in §3.4, so the audio-to-MineCLIP mapping is not independently reproducible. These issues do not disprove the methodology, but they make the headline result contingent on an unverified and possibly contaminated training/evaluation relationship.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a method called STEVE-Audio for extending the instruction-following modalities of a STEVE-1 Minecraft agent. The method consists of training an Audio-Video CLIP model for Minecraft, where audio and video embeddings are mapped into a shared latent space, and then training a prior network that maps audio embeddings from this new space into the MineCLIP visual latent space that the frozen STEVE-1 policy was trained on. The authors collect a dataset of Minecraft gameplay audio-video pairs (25 hours used for training, 600 hours to be released) and evaluate the resulting audio-conditioned agent on short-horizon item-collection tasks (wood, dirt, seeds, sand, cobblestone, leaves) against the original text- and visual-conditioned STEVE-1 baselines. The paper reports that the audio-conditioned agent generally outperforms both baselines, discusses tradeoffs between modalities, and makes training and evaluation code available.","tokens_in":9381,"tokens_out":2275,"duration_ms":23480,"significance":"The proposed methodology for extending the goal-conditioning modality of an existing generative agent without retraining its policy is a plausible and potentially useful contribution, especially because it builds on the established STEVE-1 and MineCLIP frameworks rather than introducing a new policy from scratch. If the empirical claims hold, the work demonstrates a practical route to adding audio conditioning to an embodied agent and highlights interesting modality tradeoffs. The release of training code, evaluation code, and a 600-hour Audio-Video Minecraft dataset is a concrete contribution to reproducibility and future multi-modal agent research. However, the central empirical claim is currently not supported by the presented evidence: the reported per-task ratios are mixed, there are no significance tests or confidence intervals, and the disclosed use of task-specific training data creates an uneven comparison with the baselines. These issues make the headline conclusion premature but, in my view, repairable.","major_comments":[{"comment":"The claim that the audio-conditioned agent \"generally performs better than the original STEVE-1 modalities\" is not supported by the numbers the paper itself reports. The audio agent collects 0.7x the cobblestone compared to both text and visual agents, and 0.9x the seeds compared to the visual agent; even if the other four tasks show large relative improvements, this is a mixed pattern, not a general advantage. Moreover, the evaluation uses 10 seeds per task, yet no significance tests, standard errors, or confidence intervals are reported; the percentile bars in Figure 4 give spread but do not establish statistical significance. The conclusion in Section 4.1 should be revised to state that audio conditioning performs comparably or better on a subset of tasks, and the empirical comparisons need error bars or hypothesis tests.","section":"Section 4.1, Figure 4"},{"comment":"The appendix discloses a potentially load-bearing methodological detail that undermines the comparability of the main results: the Audio-Video dataset was augmented with two hours of task-specific videos that are \"more specifically relevant to our evaluated set of tasks,\" and the text says that using YouTube data exclusively yielded an underperforming audio-conditioned agent \"(see Experimental results section below)\" -- yet no such experimental results section appears in the paper. The six evaluation tasks are exactly the item-collection behaviors that such task-specific videos presumably depict, so the Audio-Video CLIP model was trained on data from the evaluation distribution, while the text and visual baselines received no such task-specific exposure. This makes the reported audio-conditioned gains potentially attributable to training-data contamination of the evaluation setup rather than to the audio modality per se. The missing control experiment (audio conditioning trained on YouTube-only data and evaluated on the same tasks) must be presented, and the main text must state the augmentation transparently and discuss its effect on the comparison.","section":"Appendix A, Dataset and Training"},{"comment":"The training procedure for the audio prior is not specified sufficiently for reproducibility or for assessing whether the mapping is well-posed. The text says the prior is a CV AE with two-layer MLP encoder/decoder, but it does not specify the training objective (e.g., reconstruction loss, KL term, whether it is supervised with corresponding visual MineCLIP embeddings), the paired data used to train it, the batch size, learning rate, number of epochs, or how overfitting is controlled. The prior is load-bearing because it is the only component that translates the new Audio-Video CLIP latent space into the MineCLIP space that the frozen policy understands; if this mapping is poorly aligned for the evaluated tasks, the whole method fails. This omission also prevents independent reproduction of the central result.","section":"Section 3.4, Extending STEVE-1 to Condition on Audio Prompts"},{"comment":"The conclusion that audio prompting \"generally performs better\" than text and visual prompting is also used to support the hypothesis that audio-video correlation is higher than text-video correlation. This is an interesting but speculative explanation, and it is not directly tested. The comparison is confounded by the different training data used for the Audio-Video CLIP model (which includes task-specific videos) versus MineCLIP (which was trained on YouTube captions with noisy text). To support the correlation hypothesis, the authors would need to show that the audio-conditioned advantage persists when the Audio-Video CLIP is trained only on the same type of YouTube data used for MineCLIP. As written, the cross-modal comparison is not controlled.","section":"Section 4.2.1, Versatility VS. Performance"}],"minor_comments":[{"comment":"The paper alternately uses the terms \"Audio-Video CLIP\" and \"Audio-Visual CLIP\" (e.g., in the abstract vs. Section 3.1 and Figure 2). Please standardize the terminology throughout.","section":"Section 3.1"},{"comment":"The section header \"V ersatility VS. Performance\" contains an internal space; it should read \"Versatility vs. Performance.\"","section":"Section 4.2.1 header"},{"comment":"The dataset metrics table lists \"Number of test samples\" but leaves the cell empty. Please provide this value or explain why test samples are not defined.","section":"Appendix A, Table 1"},{"comment":"The number of epochs appears twice in Table 2 (once as \"Number of epochs\" and once as \"epochs\"). Remove the duplicate entry.","section":"Section 3.3 and Table 2"},{"comment":"The sentence \"A VLEN [20]introduces a multi-modal hierarchical reinforcement learning agent...\" has a missing space before \"introduces\" and an extra space after the citation bracket. Fix the typographical error.","section":"Section 2, last paragraph"},{"comment":"The paper does not discuss the computational cost of training the Audio-Video CLIP model and the prior, nor the inference-time cost of audio conditioning. A brief statement about training time and hardware would help readers assess the practical feasibility of the approach.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is, in substance, a workshop-style demonstration of an interesting extension to STEVE-1, but the current version has a missing control experiment and an overly strong central claim. The two-hour task-specific dataset augmentation is disclosed in the appendix, yet the consequences for the evaluation are not confronted in the main text; the referenced 'Experimental results section' for the YouTube-only failure is absent, which makes the claim impossible to verify. With the addition of the missing control, a proper statistical treatment, and a more measured conclusion, the paper could become a solid contribution. The open-source release of code and data is a genuine strength and should be preserved in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The concrete new thing is real: an Audio-Video CLIP for Minecraft (frozen MineCLIP video encoder + frozen AST audio encoder, with trainable StyleGAN-style mapping networks) plus an audio-to-MineCLIP prior that lets them condition the frozen STEVE-1 policy on audio without retraining it. They also release code and data. That is a legitimate extension within the STEVE-1 program, and the tradeoff discussion (audio ambiguous for place tasks, less prompt engineering needed) is sensible.\n\nThe paper does a few things well. The architecture is clearly described, the evaluation setup follows Baker/Lifshitz, and they are upfront in the appendix that YouTube-only data underperformed and that they added two hours of task-specific gameplay. That disclosure is more honest than most.\n\nThe soft spots are real but not fatal. First, Section 4.1's claim that the audio agent 'generally performs better' overstates the numbers: cobblestone is 0.7x versus both text and visual, and seeds is 0.9x versus visual. That is 'better on most tasks', not 'generally better'. Second, there are no significance tests or standard errors, and with 10 seeds some of those ratios could be noise. Third, the prior's training objective is underspecified—architecture is given (CV AE, two-layer MLPs, hidden 256, layer norm), but not the loss or what paired data it was trained on, so the mapping is not reproducible as written. Fourth and most important: the task-specific augmentation in Appendix A overlaps with the exact six evaluation tasks. The audio and video encoders saw task-distribution videos during CLIP pretraining; the text and visual baselines didn't. That makes the comparison uneven, and the appendix refers to an 'Experimental results section' showing the YouTube-only failure that does not appear anywhere in the paper. So the key control is referenced but missing.\n\nNone of this kills the methodology. The core idea—map a new modality into the existing latent goal space—is sound, and the released artifacts would let others verify. But the headline result is contingent until we see the YouTube-only numbers and a proper statistical comparison.\n\nWho should read it: people building instruction-following agents in Minecraft or extending CLIP-conditioned policies to new modalities. It would be a useful workshop paper, and with the missing details filled in it could be a solid short conference paper.\n\nFor peer review: I'd send it out rather than desk reject. The method is clear, the contribution is identifiable, and the flaws are fixable. The reviewer should ask for the prior training details, significance tests, and the YouTube-only control results.","headline":"A credible incremental extension of STEVE-1 to audio conditioning, but the headline performance claim is overstated and the missing YouTube-only control makes the main result contingent.","tokens_in":9969,"tokens_out":3220,"would_cite":false,"duration_ms":28821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding audio as a goal modality lets a Minecraft agent beat its text- and video-prompted counterparts on short-horizon collection tasks, without retraining the underlying policy.","keywords":["Minecraft","Audio","Robots","Control","CLIP","Multi-modal","Goal conditioning","Latent space mapping"],"falsifier":"Remove the two hours of task-specific gameplay from the Audio-Video dataset and rerun the same six item-collection evaluations; the paper reports this version underperforms, so a replication that finds no drop would falsify the claimed dependence on task-specific data, while a replication that reproduces the drop would confirm the result rests on that augmentation.","tokens_in":8875,"feed_emoji":"🔊","tokens_out":9739,"duration_ms":81696,"temperature":0.7,"pith_summary":"This paper claims that an existing instruction-following Minecraft agent can learn a brand-new goal modality—audio—without retraining its behavior policy. The method is to train a new audio-video CLIP model that aligns sounds with video frames, then train a prior network that translates the resulting audio embeddings into the visual goal space the agent already understands. Evaluated on six short-horizon item-collection tasks, the audio-conditioned agent collects more wood, dirt, sand, seeds, and leaves than the original text- and video-conditioned versions of the same agent, while falling short on cobblestone and on \"placing\" tasks where sounds are ambiguous. The point matters because it suggests goal conditioning is portable across modalities: new senses can be added to a frozen agent by learning a translation into its latent goal space, making multi-modal control a matter of prompt choice rather than policy retraining.","feed_headline":"Audio-conditioned Minecraft agent outperforms text and video versions","feed_subtitle":"A new audio-to-goal mapping lets the frozen agent act on sounds, expanding its senses without retraining.","key_machinery":"The load-bearing object is a two-stage latent-space translator. Stage one is the Audio-Video CLIP model: a frozen MineCLIP video encoder and a frozen Audio Spectrogram Transformer feed trainable transformation networks (ten-layer mapping networks with 1024-wide hidden layers) whose outputs are aligned by a contrastive cosine-similarity objective over matching and non-matching audio-video pairs. Stage two is a conditional variational autoencoder prior, a two-layer MLP encoder-decoder with layer normalization, that takes an audio embedding from the new space and maps it into the visual MineCLIP embedding space. Because the prior reaches back to the original goal space rather than the new model's own video space, the STEVE-1 policy remains completely frozen and interprets the audio-derived goal exactly as it would a video-derived goal.","core_discovery":"The central claim is that the latent goal space of a frozen instruction-following policy is modality-agnostic enough to accept goals translated from a completely different sense. The paper constructs an Audio-Video CLIP foundation model for Minecraft by training transformation networks on top of frozen MineCLIP video features and frozen Audio Spectrogram Transformer audio features, using a contrastive objective on one-second audio-video pairs. It then trains a conditional variational autoencoder prior that maps audio embeddings from this new space into the visual MineCLIP embedding space that the STEVE-1 policy was trained to follow. In the six item-collection tasks, the resulting audio-conditioned agent collects 1.6 to 17.7 times more of several resources than the text- and visual-conditioned baselines and loses clearly only on cobblestone and on placing tasks whose audio prompts are ambiguous. The paper presents this as evidence that audio can be a competitive or superior prompting modality and that the tradeoffs between modalities justify agents that can switch among them.","pith_inferences":["Going beyond the paper: the same two-step recipe should transfer to any new modality that co-occurs with video—thermal, depth, tactile, or electromagnetic signatures—provided a paired dataset can be collected; the bottleneck will likely be the prior's alignment quality, not the policy.","The paper's explanation that audio-video pairs are more correlated than text-video pairs predicts that tasks with highly salient sounds gain the most from audio conditioning, a ranking that could be tested by correlating per-task performance with audio distinctiveness.","The appendix's finding that YouTube-only data failed until two hours of task-specific gameplay was added suggests the alignment is data-sensitive; a scaling study over task-specific data volume would show whether the method's success is a data curation effect.","A testable combination: prompting with audio for collection tasks and text for placing tasks should outperform any single modality on the ambiguous tasks, directly testing the paper's tradeoff argument."],"forward_implications":["Existing instruction-following agents can be extended to new modalities by pairing a new CLIP model with a learned prior, leaving the policy frozen and requiring no new behavior demonstrations for the new sense.","Audio prompting can demand less prompt engineering than text or images, since sounds of a task are less variable across demonstrations than written prompts or background-cluttered video frames.","Some tasks cannot be conveyed by audio alone; placing tasks whose sounds resemble digging or generic material sounds are systematically harder for the audio-conditioned agent.","A single multi-modal agent can pick the best modality per instruction, using audio for collection-style tasks and text for abstract or complex instructions.","Because the model, code, and a 600-hour audio-video dataset are open-sourced, the recipe can be tried in other domains without rebuilding the infrastructure."],"supporting_citations":[{"why":"Supplies the frozen STEVE-1 policy and the visual goal space this work extends to audio.","marker":"[4]"},{"why":"Provides the MineCLIP video encoder and the visual MineCLIP embedding space used as the prior's target.","marker":"[6]"},{"why":"Defines the contrastive CLIP objective and cosine similarity used to train the Audio-Video CLIP model.","marker":"[5]"},{"why":"Provides the frozen Audio Spectrogram Transformer whose logits serve as audio embeddings.","marker":"[32]"},{"why":"Provides the VPT policy behind STEVE-1 and the programmatic evaluation methodology for the tasks.","marker":"[3]"},{"why":"Supplies the mapping network architecture that the trainable transformation networks are upscaled versions of.","marker":"[31]"},{"why":"Provides the variational autoencoder formulation used to build the audio-to-visual prior.","marker":"[29]"}],"fun_headline_variants":["Audio-conditioned Minecraft agent beats text and video prompts","Sound-guided agent outperforms text and visual versions in Minecraft","Frozen agent learns audio goals to expand senses in Minecraft","Audio-to-goal mapping boosts agent performance in Minecraft","New audio modality makes Minecraft agent as good as text and vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that audio embeddings from the new model can be translated into visual goal embeddings the frozen policy treats as real goals—a premise the appendix shows failed for YouTube-only data until two hours of task-specific gameplay was added.","fun_headline_variants_meta":{"raw":{"variants":["Audio-conditioned Minecraft agent beats text and video prompts","Sound-guided agent outperforms text and visual versions in Minecraft","Frozen agent learns audio goals to expand senses in Minecraft","Audio-to-goal mapping boosts agent performance in Minecraft","New audio modality makes Minecraft agent as good as text and vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00036,"raw_usage":{"total_tokens":1930,"prompt_tokens":915,"completion_tokens":1015,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":949}},"tokens_in":531,"tokens_out":1015,"duration_ms":9764,"temperature":1.0,"reasoning_tokens":949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:49:59.905042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove the two hours of task-specific gameplay from the Audio-Video dataset and rerun the same six item-collection evaluations; the paper reports this version underperforms, so a replication that finds no drop would falsify the claimed dependence on task-specific data, while a replication that reproduces the drop would confirm the result rests on that augmentation.","supporting_citations":[{"cited_title":"Lifshitz, K","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen STEVE-1 policy and the visual goal space this work extends to audio."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MineCLIP video encoder and the visual MineCLIP embedding space used as the prior's target."},{"cited_title":"Karras, M","cited_arxiv_id":null,"evidence_quote":"Supplies the mapping network architecture that the trainable transformation networks are upscaled versions of."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the variational autoencoder formulation used to build the audio-to-visual prior."}],"review_version":1}