REVIEW 5 cited by
Learning to Model the World with Language
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
To interact with humans and act in the world, agents need to understand the range of language that people use and relate it to the visual world. While current agents can learn to execute simple language instructions, we aim to build agents that leverage diverse language -- language like "this button turns on the TV" or "I put the bowls away" -- that conveys general knowledge, describes the state of the world, provides interactive feedback, and more. Our key idea is that agents should interpret such diverse language as a signal that helps them predict the future: what they will observe, how the world will behave, and which situations will be rewarded. This perspective unifies language understanding with future prediction as a powerful self-supervised learning objective. We instantiate this in Dynalang, an agent that learns a multimodal world model to predict future text and image representations, and learns to act from imagined model rollouts. While current methods that learn language-conditioned policies degrade in performance with more diverse types of language, we show that Dynalang learns to leverage environment descriptions, game rules, and instructions to excel on tasks ranging from game-playing to navigating photorealistic home scans. Finally, we show that our method enables additional capabilities due to learning a generative model: Dynalang can be pretrained on text-only data, enabling learning from offline datasets, and generate language grounded in an environment.
Forward citations
Cited by 5 Pith papers
-
LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratory
A chemical-lab failure sim, 20K-trajectory dataset, six-axis benchmark, and specialized VLM raise failure detection to 90.8% on seen scenes and lift downstream policy success by 4–16 points.
-
Audio-Visual World Models: Learning Physically Grounded Multisensory Dynamics
Future video frames, binaural audio, and task rewards can be generated jointly under action control by a diffusion transformer trained on a new 30-hour simulated audio-visual benchmark, with small downstream navigation gains.
-
Towards Language-Augmented Multi-Agent Deep Reinforcement Learning
Giving MARL agents supervised language targets for describing observations improves task performance, representation structure, and robustness to new teammates compared to emergent communication baselines.
-
Evaluating LLMs as Interpretable Controllers for Dynamical Systems
In a simulated thermal enclosure, high-capability LLMs achieve accurate setpoint tracking and coherent reasoning, while small models fail; giving the LLM the exact plant simulator as a prediction tool improves smoothn...
-
Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation
A dual-branch text-imagination system, with one LLM branch for state estimation and one for candidate-direction description, improves R2R navigation success over prior LLM-based VLN methods.
Discussion (0). Sign in to comment.