REVIEW 6 cited by
Scaling Instructable Agents Across Many Simulated Worlds
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Building embodied AI systems that can follow arbitrary language instructions in any 3D environment is a key challenge for creating general AI. Accomplishing this goal requires learning to ground language in perception and embodied actions, in order to accomplish complex tasks. The Scalable, Instructable, Multiworld Agent (SIMA) project tackles this by training agents to follow free-form instructions across a diverse range of virtual 3D environments, including curated research environments as well as open-ended, commercial video games. Our goal is to develop an instructable agent that can accomplish anything a human can do in any simulated 3D environment. Our approach focuses on language-driven generality while imposing minimal assumptions. Our agents interact with environments in real-time using a generic, human-like interface: the inputs are image observations and language instructions and the outputs are keyboard-and-mouse actions. This general approach is challenging, but it allows agents to ground language across many visually complex and semantically rich environments while also allowing us to readily run agents in new environments. In this paper we describe our motivation and goal, the initial progress we have made, and promising preliminary results on several diverse research environments and a variety of commercial video games.
Forward citations
Cited by 6 Pith papers
-
Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time
The manuscript body introduces PokeGym, a vision-only automated 3D-game benchmark, while the abstract claims a G-EvoMAC method and 60.18% success rate absent from the body.
-
Update-Free On-Policy Steering via Verifiers
Lightweight verifiers trained on a diffusion policy’s own evaluation rollouts raise real-robot success rates ~49% on average via Best-of-N or classifier guidance, without changing base parameters.
-
BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
BIMgent, a GUI-controlling LLM agent, completes 32% of BIM building modeling tasks end-to-end, outperforming baseline computer-use agents that complete none.
-
Generative Physical AI in Vision: A Survey
A structured review that categorizes physics-aware generative models in vision into explicit-simulation and implicit-learning families and proposes six integration paradigms.
-
Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning
A lightweight attention module that weights embeddings from multiple pre-trained models achieves comparable Atari RL performance to end-to-end training, with improved robustness to visual changes.
-
AGI Requires a Coordination Layer on Top of Pattern Repositories
The paper claims AGI requires a System-2 coordination layer on top of LLM pattern repositories, formalized by an anchoring score S = ρd − dr − γ log k.
Discussion (0). Continue with ORCID to comment.