Pith. sign in

REVIEW 4 major objections 6 minor 34 references

STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding audio as a goal modality lets a Minecraft agent beat its text- and video-prompted counterparts on short-horizon collection tasks, without retraining the underlying policy.

desk verdict A credible incremental extension of STEVE-1 to audio conditioning, but the headline performance claim is overstated and the missing YouTube-only control makes the main result contingent. read the letter →

arxiv 2412.00949 v1 pith:XJ7MT7PZ submitted 2024-12-01 cs.LG cs.AIcs.RO

Nicholas Lenzen , Amogh Raut , Andrew Melnik This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords MinecraftAudioRobotsControlCLIPMulti-modalGoalconditioningLatentspacemapping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an existing instruction-following Minecraft agent can learn a brand-new goal modality—audio—without retraining its behavior policy. The method is to train a new audio-video CLIP model that aligns sounds with video frames, then train a prior network that translates the resulting audio embeddings into the visual goal space the agent already understands. Evaluated on six short-horizon item-collection tasks, the audio-conditioned agent collects more wood, dirt, sand, seeds, and leaves than the original text- and video-conditioned versions of the same agent, while falling short on cobblestone and on "placing" tasks where sounds are ambiguous. The point matters because it suggests goal conditioning is portable across modalities: new senses can be added to a frozen agent by learning a translation into its latent goal space, making multi-modal control a matter of prompt choice rather than policy retraining.

What carries the argument

The load-bearing object is a two-stage latent-space translator. Stage one is the Audio-Video CLIP model: a frozen MineCLIP video encoder and a frozen Audio Spectrogram Transformer feed trainable transformation networks (ten-layer mapping networks with 1024-wide hidden layers) whose outputs are aligned by a contrastive cosine-similarity objective over matching and non-matching audio-video pairs. Stage two is a conditional variational autoencoder prior, a two-layer MLP encoder-decoder with layer normalization, that takes an audio embedding from the new space and maps it into the visual MineCLIP embedding space. Because the prior reaches back to the original goal space rather than the new model's own video space, the STEVE-1 policy remains completely frozen and interprets the audio-derived goal exactly as it would a video-derived goal.

What would settle it

Remove the two hours of task-specific gameplay from the Audio-Video dataset and rerun the same six item-collection evaluations; the paper reports this version underperforms, so a replication that finds no drop would falsify the claimed dependence on task-specific data, while a replication that reproduces the drop would confirm the result rests on that augmentation.

Watch

Extended reading notes

Core claim

The central claim is that the latent goal space of a frozen instruction-following policy is modality-agnostic enough to accept goals translated from a completely different sense. The paper constructs an Audio-Video CLIP foundation model for Minecraft by training transformation networks on top of frozen MineCLIP video features and frozen Audio Spectrogram Transformer audio features, using a contrastive objective on one-second audio-video pairs. It then trains a conditional variational autoencoder prior that maps audio embeddings from this new space into the visual MineCLIP embedding space that the STEVE-1 policy was trained to follow. In the six item-collection tasks, the resulting audio-conditioned agent collects 1.6 to 17.7 times more of several resources than the text- and visual-conditioned baselines and loses clearly only on cobblestone and on placing tasks whose audio prompts are ambiguous. The paper presents this as evidence that audio can be a competitive or superior prompting modality and that the tradeoffs between modalities justify agents that can switch among them.

Load-bearing premise

The load-bearing premise is that audio embeddings from the new model can be translated into visual goal embeddings the frozen policy treats as real goals—a premise the appendix shows failed for YouTube-only data until two hours of task-specific gameplay was added.

Editorial extensions

If this is right

  • Existing instruction-following agents can be extended to new modalities by pairing a new CLIP model with a learned prior, leaving the policy frozen and requiring no new behavior demonstrations for the new sense.
  • Audio prompting can demand less prompt engineering than text or images, since sounds of a task are less variable across demonstrations than written prompts or background-cluttered video frames.
  • Some tasks cannot be conveyed by audio alone; placing tasks whose sounds resemble digging or generic material sounds are systematically harder for the audio-conditioned agent.
  • A single multi-modal agent can pick the best modality per instruction, using audio for collection-style tasks and text for abstract or complex instructions.
  • Because the model, code, and a 600-hour audio-video dataset are open-sourced, the recipe can be tried in other domains without rebuilding the infrastructure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: the same two-step recipe should transfer to any new modality that co-occurs with video—thermal, depth, tactile, or electromagnetic signatures—provided a paired dataset can be collected; the bottleneck will likely be the prior's alignment quality, not the policy.
  • The paper's explanation that audio-video pairs are more correlated than text-video pairs predicts that tasks with highly salient sounds gain the most from audio conditioning, a ranking that could be tested by correlating per-task performance with audio distinctiveness.
  • The appendix's finding that YouTube-only data failed until two hours of task-specific gameplay was added suggests the alignment is data-sensitive; a scaling study over task-specific data volume would show whether the method's success is a data curation effect.
  • A testable combination: prompting with audio for collection tasks and text for placing tasks should outperform any single modality on the ambiguous tasks, directly testing the paper's tradeoff argument.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a method called STEVE-Audio for extending the instruction-following modalities of a STEVE-1 Minecraft agent. The method consists of training an Audio-Video CLIP model for Minecraft, where audio and video embeddings are mapped into a shared latent space, and then training a prior network that maps audio embeddings from this new space into the MineCLIP visual latent space that the frozen STEVE-1 policy was trained on. The authors collect a dataset of Minecraft gameplay audio-video pairs (25 hours used for training, 600 hours to be released) and evaluate the resulting audio-conditioned agent on short-horizon item-collection tasks (wood, dirt, seeds, sand, cobblestone, leaves) against the original text- and visual-conditioned STEVE-1 baselines. The paper reports that the audio-conditioned agent generally outperforms both baselines, discusses tradeoffs between modalities, and makes training and evaluation code available.

Significance. The proposed methodology for extending the goal-conditioning modality of an existing generative agent without retraining its policy is a plausible and potentially useful contribution, especially because it builds on the established STEVE-1 and MineCLIP frameworks rather than introducing a new policy from scratch. If the empirical claims hold, the work demonstrates a practical route to adding audio conditioning to an embodied agent and highlights interesting modality tradeoffs. The release of training code, evaluation code, and a 600-hour Audio-Video Minecraft dataset is a concrete contribution to reproducibility and future multi-modal agent research. However, the central empirical claim is currently not supported by the presented evidence: the reported per-task ratios are mixed, there are no significance tests or confidence intervals, and the disclosed use of task-specific training data creates an uneven comparison with the baselines. These issues make the headline conclusion premature but, in my view, repairable.

major comments (4)
  1. [Section 4.1, Figure 4] The claim that the audio-conditioned agent "generally performs better than the original STEVE-1 modalities" is not supported by the numbers the paper itself reports. The audio agent collects 0.7x the cobblestone compared to both text and visual agents, and 0.9x the seeds compared to the visual agent; even if the other four tasks show large relative improvements, this is a mixed pattern, not a general advantage. Moreover, the evaluation uses 10 seeds per task, yet no significance tests, standard errors, or confidence intervals are reported; the percentile bars in Figure 4 give spread but do not establish statistical significance. The conclusion in Section 4.1 should be revised to state that audio conditioning performs comparably or better on a subset of tasks, and the empirical comparisons need error bars or hypothesis tests.
  2. [Appendix A, Dataset and Training] The appendix discloses a potentially load-bearing methodological detail that undermines the comparability of the main results: the Audio-Video dataset was augmented with two hours of task-specific videos that are "more specifically relevant to our evaluated set of tasks," and the text says that using YouTube data exclusively yielded an underperforming audio-conditioned agent "(see Experimental results section below)" -- yet no such experimental results section appears in the paper. The six evaluation tasks are exactly the item-collection behaviors that such task-specific videos presumably depict, so the Audio-Video CLIP model was trained on data from the evaluation distribution, while the text and visual baselines received no such task-specific exposure. This makes the reported audio-conditioned gains potentially attributable to training-data contamination of the evaluation setup rather than to the audio modality per se. The missing control experiment (audio conditioning trained on YouTube-only data and evaluated on the same tasks) must be presented, and the main text must state the augmentation transparently and discuss its effect on the comparison.
  3. [Section 3.4, Extending STEVE-1 to Condition on Audio Prompts] The training procedure for the audio prior is not specified sufficiently for reproducibility or for assessing whether the mapping is well-posed. The text says the prior is a CV AE with two-layer MLP encoder/decoder, but it does not specify the training objective (e.g., reconstruction loss, KL term, whether it is supervised with corresponding visual MineCLIP embeddings), the paired data used to train it, the batch size, learning rate, number of epochs, or how overfitting is controlled. The prior is load-bearing because it is the only component that translates the new Audio-Video CLIP latent space into the MineCLIP space that the frozen policy understands; if this mapping is poorly aligned for the evaluated tasks, the whole method fails. This omission also prevents independent reproduction of the central result.
  4. [Section 4.2.1, Versatility VS. Performance] The conclusion that audio prompting "generally performs better" than text and visual prompting is also used to support the hypothesis that audio-video correlation is higher than text-video correlation. This is an interesting but speculative explanation, and it is not directly tested. The comparison is confounded by the different training data used for the Audio-Video CLIP model (which includes task-specific videos) versus MineCLIP (which was trained on YouTube captions with noisy text). To support the correlation hypothesis, the authors would need to show that the audio-conditioned advantage persists when the Audio-Video CLIP is trained only on the same type of YouTube data used for MineCLIP. As written, the cross-modal comparison is not controlled.
minor comments (6)
  1. [Section 3.1] The paper alternately uses the terms "Audio-Video CLIP" and "Audio-Visual CLIP" (e.g., in the abstract vs. Section 3.1 and Figure 2). Please standardize the terminology throughout.
  2. [Section 4.2.1 header] The section header "V ersatility VS. Performance" contains an internal space; it should read "Versatility vs. Performance."
  3. [Appendix A, Table 1] The dataset metrics table lists "Number of test samples" but leaves the cell empty. Please provide this value or explain why test samples are not defined.
  4. [Section 3.3 and Table 2] The number of epochs appears twice in Table 2 (once as "Number of epochs" and once as "epochs"). Remove the duplicate entry.
  5. [Section 2, last paragraph] The sentence "A VLEN [20]introduces a multi-modal hierarchical reinforcement learning agent..." has a missing space before "introduces" and an extra space after the citation bracket. Fix the typographical error.
  6. [General] The paper does not discuss the computational cost of training the Audio-Video CLIP model and the prior, nor the inference-time cost of audio conditioning. A brief statement about training time and hardware would help readers assess the practical feasibility of the approach.

Circularity Check

1 steps flagged · score 4.0 of 10

The audio-conditioned gain in §4.1 is partly fitted to the evaluation suite: Appendix A discloses task-specific training data added after YouTube-only failure, and the promised control experiment is missing.

  1. fitted input called prediction [Appendix A ('Dataset and Training'); Sections 3.5/4.1]
    "We also augmented our Audio-Video dataset with short 10-20 minute videos that are more specifically relevant to our evaluated set of tasks (an additional two hours in total). ... but including a short amount of more task-specific data significantly improved performance. ... These results indicate that our audio-conditioned agent generally performs better than the original STEVE-1 modalities"

    The reported headline result is not an independent prediction of the proposed modality-extension method. The paper states that the Audio-Video dataset was augmented with task-specific videos because YouTube-only training yielded an underperforming agent, and the augmentation was chosen to be 'more specifically relevant to our evaluated set of tasks.' The evaluation in §3.5/§4.1 uses exactly those short-horizon item-collection tasks (dirt, wood, seeds, sand, cobblestone, leaves).

full rationale

The technical derivation chain is otherwise self-contained: the Audio-Video CLIP model is trained with frozen MineCLIP and AST encoders plus learned transformation networks, the audio prior maps into the MineCLIP visual latent space, and the STEVE-1 policy is frozen and externally sourced. No equation reduces the result to its inputs by construction, and the self-citations to Melnik et al. (e.g., the StyleGAN mapping network reference) are not load-bearing. The main circularity concern is the evaluation-selection loop disclosed in Appendix A: the audio model's training data was adjusted using the evaluation task family after YouTube-only data underperformed, and the paper then reports superiority on those same tasks as a general finding. This makes the §4.1 claim partly fitted to the evaluation set rather than an out-of-sample prediction. The missing referenced 'Experimental results section below' further prevents verification of the control. Because the policy, MineCLIP, and the core mapping methodology have independent content, the paper is not wholly circular; score 4 reflects partial, not complete, circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on domain assumptions about modality correlation and latent-space alignability. No new physical entities are introduced. The free parameters are empirical design choices and, most importantly, the two hours of task-specific training data added after observing poor YouTube-only performance.

free parameters (5)
  • Task-specific training data (hours) = 2 hours
    Appendix A states the authors added 10 to 20 minute task-specific videos because YouTube-only training yielded an underperforming agent. This post hoc data choice directly shapes the reported evaluation scores.
  • Audio-video CLIP training epochs = 100
    Table 2 lists 100 epochs and a fixed learning rate; no ablation demonstrates that this schedule is optimal or that the result is robust to it.
  • Transformation network size = 10 layers, 1024 hidden units
    Section 3.1 says this upscaled StyleGAN3 mapping network was best-performing, but no supporting ablation is shown.
  • Audio-video sample window and overlap = 1 second, 75% overlap
    Appendix A specifies these dataset construction choices without ablations; they determine what audio-video pairs the CLIP model sees.
  • Prior hidden dimension = 256
    Section 3.4 sets the CVAE prior's hidden size to 256 by analogy with STEVE-1, but the prior's training data and objective are not specified.
assumptions (3)
  • domain assumption Audio and video from unlabeled Minecraft YouTube gameplay are correlated strongly enough to learn goal-relevant audio embeddings.
    Invoked in Sections 3.2 and 4.2.1; Appendix A shows this fails for the original YouTube-only set, requiring task-specific augmentation.
  • domain assumption Frozen AST and MineCLIP encoders, with lightweight transformation networks, can form a shared audio-video latent space.
    Assumed in Section 3.1; no experiments vary the encoders or the transformation network capacity.
  • domain assumption A learned prior can translate audio embeddings from the new latent space into the MineCLIP visual goal space used by the frozen STEVE-1 policy.
    Central to Section 3.4; the prior's training procedure is not described, so this mapping is an unverified premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft." pith.science (2026). https://pith.science/paper/XJ7MT7PZ

@misc{pith2026241200949,
  author       = {Pith},
  title        = {Pith review of: STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJ7MT7PZ}},
  note         = {Machine review of arXiv:2412.00949}
}
read the original abstract

Recently, the STEVE-1 approach has been introduced as a method for training generative agents to follow instructions in the form of latent CLIP embeddings. In this work, we present a methodology to extend the control modalities by learning a mapping from new input modalities to the latent goal space of the agent. We apply our approach to the challenging Minecraft domain, and extend the goal conditioning to include the audio modality. The resulting audio-conditioned agent is able to perform on a comparable level to the original text-conditioned and visual-conditioned agents. Specifically, we create an Audio-Video CLIP foundation model for Minecraft and an audio prior network which together map audio samples to the latent goal space of the STEVE-1 policy. Additionally, we highlight the tradeoffs that occur when conditioning on different modalities. Our training code, evaluation code, and Audio-Video CLIP foundation model for Minecraft are made open-source to help foster further research into multi-modal generalist sequential decision-making agents.

Figures

Figures reproduced from arXiv: 2412.00949 by the authors.

Figure 1
Figure 1. Examples of evaluation tasks within the Minecraft environment, showcasing the observed [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Our architecture for the Audio-Video CLIP model learns a shared latent space by jointly [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Our architecture for audio prompting of the STEVE-1 agent [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance comparison of audio-conditioned STEVE-1 (created using our proposed [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages

  1. [1]

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021

    work page 2021

  2. [2]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    arXiv 2023

  3. [3]

    Baker, I

    B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022

    2022

  4. [4]

    Lifshitz, K

    S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. McIlraith. Steve-1: A generative model for text-to-behavior in minecraft. 2023

    work page 2023

  5. [5]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  6. [6]

    L. Fan, G. Wang, Y . Jiang, A. Mandlekar, Y . Yang, H. Zhu, A. Tang, D.-A. Huang, Y . Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=rc8o_j8I8PX

    work page 2022

  7. [7]

    W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440, 2019

    arXiv 1907

  8. [8]

    Behavioral Cloning via Search in Video PreTraining Latent Space

    F. Malato, F. Leopold, A. Raut, V . Hautam¨aki, and A. Melnik. Behavioral cloning via search in video pretraining latent space. arXiv preprint arXiv:2212.13326, 2022

    work page Pith review arXiv 2022

Show all 34 references
  1. [9]

    Malato, F

    F. Malato, F. Leopold, A. Melnik, and V . Hautam ¨aki. Zero-shot imitation policy via search in demonstration dataset. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7590–7594. IEEE, 2024

    2024

  2. [10]

    Melnik, A

    A. Melnik, A. Harter, C. Limberg, K. Rana, N. S ¨underhauf, and H. Ritter. Critic guided segmentation of rewarding objects in first-person views. In KI 2021: Advances in Artificial Intelligence: 44th German Conference on AI, Virtual Event, September 27–October 1, 2021, Proceed...

    2021

  3. [11]

    Milani, A

    S. Milani, A. Kanervisto, K. Ramanauskas, S. Schulhoff, B. Houghton, S. Mohanty, B. Gal- braith, K. Chen, Y . Song, T. Zhou, et al. Towards solving fuzzy tasks with human feedback: A retrospective of the minerl basalt 2022 competition. arXiv preprint arXiv:2303.13512, 2023

    2022 arXiv

  4. [12]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandku- mar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  5. [13]

    Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, and Y . Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023

    2023 arXiv

  6. [14]

    Z. Wang, S. Cai, A. Liu, Y . Jin, J. Hou, B. Zhang, H. Lin, Z. He, Z. Zheng, Y . Yang, et al. Jarvis- 1: Open-world multi-task agents with memory-augmented multimodal language models.arXiv preprint arXiv:2311.05997, 2023

    2023 arXiv

  7. [15]

    S. Cai, B. Zhang, Z. Wang, X. Ma, A. Liu, and Y . Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235, 2023. 9

    2023 arXiv

  8. [16]

    E. Zhou, Y . Qin, Z. Yin, Y . Huang, R. Zhang, L. Sheng, Y . Qiao, and J. Shao. Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control. arXiv preprint arXiv:2403.12037, 2024

    2024 arXiv

  9. [17]

    Mineflayer, 2011

    PrismarineJS and Others. Mineflayer, 2011. URL https://github.com/PrismarineJS/ mineflayer

    2011

  10. [18]

    Y . Qin, E. Zhou, Q. Liu, Z. Yin, L. Sheng, R. Zhang, Y . Qiao, and J. Shao. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. arXiv preprint arXiv:2312.07472, 2023

    2023 arXiv

  11. [19]

    R. Shah, R. Mart ´ın-Mart´ın, and Y . Zhu. Mutex: Learning unified policies from multimodal task specifications. In 7th Annual Conference on Robot Learning , 2023. URL https:// openreview.net/forum?id=PwqiqaaEzJ

    2023

  12. [20]

    S. Paul, A. K. Roy-Chowdhury, and A. Cherian. Avlen: Audio-visual-language embodied navigation in 3d environments. arXiv preprint arXiv:2210.07940, 2022

    2022 arXiv

  13. [21]

    Guzhov, F

    A. Guzhov, F. Raue, J. Hees, and A. Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976–980. IEEE, 2022

    2022

  14. [22]

    L. Ruan, A. Hu, Y . Song, L. Zhang, S. Zheng, and Q. Jin. Accommodating audio modal- ity in clip for multimodal processing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9641–9649, 2023

    2023

  15. [23]

    K. Rana, A. Melnik, and N. S ¨underhauf. Contrastive language, action, and state pre-training for robot learning. arXiv preprint arXiv:2304.10782, 2023

    2023 arXiv

  16. [24]

    Ramesh, P

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  17. [25]

    N. Bach, A. Melnik, M. Schilling, T. Korthals, and H. Ritter. Learn to move through a combi- nation of policy gradient algorithms: Ddpg, d4pg, and td3. InMachine Learning, Optimization, and Data Science: 6th International Conference, LOD 2020, Siena, Italy, July 19–23, 2020, R...

    2020

  18. [26]

    X. Dang, S. Edelkamp, and N. Ribault. Clip-motion: Learning reward functions for robotic actions using consecutive observations. arXiv preprint arXiv:2311.03485, 2023

    2023 arXiv

  19. [27]

    Baumli, S

    K. Baumli, S. Baveja, F. Behbahani, H. Chan, G. Comanici, S. Flennerhag, M. Gazeau, K. Hol- sheimer, D. Horgan, M. Laskin, et al. Vision-language models as a source of rewards. arXiv preprint arXiv:2312.09187, 2023

    2023 arXiv

  20. [28]

    Sontakke, J

    S. Sontakke, J. Zhang, S. Arnold, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti. Robo- clip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36, 2024

    2024

  21. [29]

    D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2022

    2022

  22. [30]

    K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep condi- tional generative models. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett, editors, Advances in Neural Information Processing Systems , volume 28. Curran As- sociates, Inc...

    2015

  23. [31]

    Karras, M

    T. Karras, M. Aittala, S. Laine, E. H ¨ark¨onen, J. Hellsten, J. Lehtinen, and T. Aila. Alias- free generative adversarial networks. In Proc. NeurIPS , 2021. URL https://github. com/NVlabs/stylegan3/blob/c233a919a6faee6e36a316ddd4eddababad1adf9/ training/networks_stylegan3.py#L69. 10

    2021

  24. [32]

    Gong, Y .-A

    Y . Gong, Y .-A. Chung, and J. Glass. Ast: Audio spectrogram transformer. InProc. Interspeech 2021, pages 571–575, 2021. doi:10.21437/Interspeech.2021-698

    2021 doi

  25. [33]

    Melnik, M

    A. Melnik, M. Miasayedzenkau, D. Makaravets, D. Pirshtuk, E. Akbulut, D. Holzmann, T. Renusch, G. Reichert, and H. Ritter. Face generation and editing with stylegan: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  26. [34]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Stochastic gradient descent with warm restarts. In Proceedings of the 5th Int. Conf. Learning Representations, pages 1–16. 11 A Dataset and Training We also augmented our Audio-Video dataset with short 10-20 minute videos that are more specif- ical...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.