PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning

Angel Villar-Corrales; Sven Behnke

REVIEW 3 major objections 5 minor 15 references

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PlaySlot learns inverse latent dynamics: from unlabeled video frames it extracts discrete-plus-continuous latent actions, and uses them to predict multiple controllable futures and to imitate robot behavior sample-efficiently.

desk verdict A well-built object-centric world model with a genuinely useful per-object latent action design, but the 'unlabeled video' promise is softer than the abstract implies because the action decoder needs real action labels. read the letter →

arxiv 2502.07600 v2 pith:TNY7MZ5E submitted 2025-02-11 cs.CV cs.RO

Angel Villar-Corrales , Sven Behnke This is my paper · ORCID

classification cs.CVcs.RO

keywords object-centriclearningvideopredictionlatentactionsinversedynamicsworldmodelsbehaviorcloningunsupervisedrepresentationslotattention

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PlaySlot claims that a robot can learn to predict and control its environment from plain video, with no action labels, by first splitting each frame into object slots and then learning an inverse dynamics module that recovers the latent action between consecutive frames. These latent actions are hybrid: each one combines a discrete action prototype with a continuous variability correction, so the model can represent distinct high-level behaviors while still interpolating fine-grained motion. A transformer-based conditional predictor then forecasts future object states and pixels from past slots, conditioned on actions that can be inferred from video, supplied by a user, or generated by a learned policy. The paper reports that PlaySlot outperforms stochastic and object-centric video prediction baselines on tasks with object interactions and multiple moving objects, and that its inferred latent actions enable sample-efficient learning of robot behaviors from unlabeled demonstrations. If correct, PlaySlot offers a way to build controllable world models from the large amount of unlabeled video already available.

What carries the argument

The load-bearing machinery is the inverse dynamics module InvDyn paired with the conditional object-centric predictor cOCVP. InvDyn takes slot sets from two consecutive frames, passes the current slots together with a learnable [ACT] token through a transformer encoder in the single-agent variant, or processes each slot with a shared MLP in the multi-agent variant, and models the scene dynamics at each time step as a Gaussian. The latent action is the difference of those distributions, $$\hat{z}_t \sim \mathcal{N}(\mu_{d_{t+1}}-\mu_{d_t},\ \$sigma^{2}$_{d_{t+1}}+\$sigma^{2}$_{d_t}),$$ which is then parameterized as the sum of a vector-quantized discrete prototype $p_t$ and a continuous variability embedding $v_t$. This hybrid parameterization funnels the action through a low-dimensional bottleneck, preventing the model from trivially encoding future frame content into the action. cOCVP is a transformer encoder that takes past slots, prototypes, and variability embeddings, adds time-step positional encodings while preserving permutation equivariance across objects, and autoregressively outputs the next slot set; the slot masks then render predicted frames. The pipeline is trained without teacher forcing, so the predictor learns to handle its own imperfect predictions.

What would settle it

Train PlaySlot on unlabeled BlockPush video, then train the small action decoder on a held-out labeled subset and measure how accurately it reconstructs ground-truth robot commands from the inferred latent actions on the remaining demonstrations; if reconstruction error is high, the latent action space does not actually capture the scene dynamics. A complementary check is to run the model on videos where two identical-looking objects cross or occlude and test whether slot identities are preserved and whether prediction error spikes relative to labeled-action baselines.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that the action needed to explain a video transition can be extracted from object-centric representations alone, without action supervision, and that this extracted action is a usable control signal. Given two consecutive frames decomposed into object slots, the InvDyn module models the distribution of scene dynamics at each of the two time steps and defines the latent action as the difference of those distributions; it then vector-quantizes that action into a discrete prototype plus a continuous variability embedding. Conditioning the object-centric predictor on these latent actions makes future generation controllable: the same initial frame branches into different futures depending on which latent action is fed in, and the branches align with semantic behaviors such as moving left, right, up, down, opening a gripper, or staying. The paper also claims the inferred actions transfer to behavior learning: a policy trained by behavior cloning to output PlaySlot's latent actions, with only a small labeled set for decoding them into executable commands, improves success rates with far fewer demonstrations than baselines that use holistic representations.

Load-bearing premise

The whole pipeline rests on the scene parser producing temporally consistent slots that bind to the same objects frame after frame; if slots drift, merge, or split, the latent actions and predicted futures inherit those errors, and the paper itself notes that its parser generalizes poorly to complex real-world scenes.

Editorial extensions

If this is right

Video prediction models can be trained without action labels by conditioning on latent actions inferred from video dynamics, which widens the applicable data to any unlabeled demonstration corpus.
A single reference frame plus a latent action—whether inferred, user-supplied, or policy-generated—yields multiple semantically distinct future rollouts from the same model.
Per-object latent actions scale to scenes with several independently moving agents, whereas a single global action vector degrades sharply as the number of objects grows.
Behavior cloning can operate entirely in a learned latent action space, needing only a small action decoder for execution, so sample efficiency improves with object-centric rather than holistic representations.
The same inverse dynamics module can annotate unlabeled demonstration datasets with latent actions, which then serve as supervision for downstream policy learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension, which the paper itself points toward, is swapping the scene parser for a more capable object-centric model: since InvDyn and cOCVP operate on slots, a stronger decomposition method could inherit the framework and extend it to complex real-world scenes.
Because latent actions are defined as differences of per-frame dynamics distributions, the same InvDyn module could act as a general-purpose action-annotation engine for other vision or robotics pipelines that currently require ground-truth action labels.
A direct test of the action space's quality would measure how much information the inferred latent actions carry about ground-truth commands: training the action decoder on a small labeled set and evaluating on held-out demonstrations would quantify whether the latent actions capture the true scene dynamics or only appearance-level differences.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the core idea is good: PlaySlot couples slot-based scene parsing with an inverse dynamics module that infers latent actions from unlabeled video, then uses those latent actions to condition an object-centric predictor. The per-object variant (InvDynM) is the most interesting piece—it scales cleanly to multiple moving objects, which the holistic baselines don't. Second, the 'unlabeled video' claim is stretched. The learned policy is trained on latent actions from unlabeled video, but the success-rate numbers in Figure 7 only materialize after an action decoder Da, trained on ground-truth actions, maps latent actions to executable controls. The paper says Da uses 'a small set' of action-labeled data, but Appendix C shows the same 900 (ButtonPress) and 4500 (BlockPush) expert demonstrations are used to train both the policy and the action decoder. That's not a small set, and it means the reported behavior-learning results are not from unlabeled demonstrations alone. This is the paper's main soft spot, and it's fixable by reporting Da's labeled-data budget separately and evaluating how performance varies with it.

What the paper does well: the method is clearly specified, the ablations (Table 2, Figure 4, prototype count in Table E.1) are informative, and the appendix is honest about SA Vi's limits in complex real scenes. The hybrid VQ action space is a reasonable middle ground between discrete and continuous latents, and it checks out empirically. The code and videos are released. This is a competent, reproducible piece of work.

The other issues are more moderate. The abstract claims PlaySlot 'outperforms' baselines, but in Table 1 SVG has higher PSNR on ButtonPress (32.23 vs 26.03) and PlaySlot trails on SSIM for BlockPush (0.890 vs 0.898). The strong wins are on BlockPush LPIPS and GridShapes, not everywhere. The video-prediction protocol conditions on latent actions inferred from the ground-truth future, so the numbers measure the value of a bottlenecked future summary rather than pure forecasting; this is transparent and applies to the baselines equally, but it should be stated plainly in the main text. No error bars appear in Tables 1 and 2, which is minor but annoying.

Overall, this is a real contribution to object-centric video prediction and latent-action learning. The right audience is researchers working on object-centric world models for robotics or on unsupervised action discovery. It deserves a serious referee; my recommendation is to engage with it, but push the authors to fix the Da accounting, temper the abstract, and add error bars. The central method is sound and the limitations are honestly stated, so I'd happily read a revised version.

Referee Report

3 major / 5 minor

Summary. PlaySlot proposes an object-centric video prediction framework that parses video frames into slot representations, infers latent actions via an inverse dynamics module, and uses these latent actions to condition an autoregressive transformer predictor. The authors claim that this approach outperforms stochastic and object-centric baselines for video prediction across several environments, and that the inferred latent actions enable sample-efficient learning of robot behaviors from unlabeled video demonstrations. The paper includes experiments on BlockPush, ButtonPress, GridShapes, and real-world Sketchy data, plus behavior-learning results in simulation.

Significance. If the central claims hold, PlaySlot would be a useful contribution: it combines object-centric representations with a hybrid discrete/continuous latent action space, making the world model controllable and interpretable, and it demonstrates that inverse latent dynamics can support downstream behavior learning. The paper is clearly written, the method is well specified, and the authors provide code and videos. However, the evidence as presented does not fully support the strength of the abstract's claims: the video prediction numbers are mixed relative to baselines, the evaluation protocol conditions on ground-truth future information, and the behavior-learning results rely on an action decoder trained with ground-truth actions whose dataset size is not clarified. These are fixable issues, but they are load-bearing for the paper's main message.

major comments (3)

[Abstract / Table 1] The abstract states that PlaySlot 'outperforms both stochastic and object-centric baselines for video prediction across different environments,' but Table 1 contradicts this on ButtonPress, where SVG achieves much higher PSNR (32.23 vs 26.03) and SSIM (0.950 vs 0.878), and on BlockPush, where SVG has slightly higher SSIM (0.898 vs 0.890). On Sketchy the results are comparable. Since Tables 1 and 2 report no error bars or significance tests, the claimed overall outperformance is not established. Please report repeated-seed statistics and temper the claim to the environments and metrics where PlaySlot is actually better.
[Section 4.2] The video prediction evaluation conditions on latent actions or vectors inferred from the ground-truth future sequence, as stated in Section 4.2 ('for CADDY, PlaySlot and SVG, we predict future frames conditioned on latent actions or vectors inferred from the ground truth sequence'). The reported PSNR/SSIM/LPIPS therefore measure the model's ability to reconstruct frames from a bottlenecked summary of the future transition, not its ability to forecast from the past alone. This protocol is applied uniformly to the controllable baselines, so the comparison is fair, but the abstract and conclusions describe the result as 'video prediction' without this qualifier. Please either add an evaluation that predicts without access to future information (e.g., using the learned policy prior) or explicitly frame the numbers as conditional future reconstruction.
[Section 3.4 / Appendix C] The behavior-learning claim that policies are learned 'from unlabeled video demonstrations' is not substantiated as stated. Eq. (15) trains an action decoder Da with ground-truth actions, and Section 3.4 says Da is trained on 'a small set of action-labeled data.' However, Appendix C.1 and C.2 reveal that the full expert demonstration sets (4,500 BlockPush sequences and 900 ButtonPress sequences) are used to train both the policy model and the action decoder. Since the reported success rates in Figs. 7 and 9 rely on executing decoded latent actions in the simulator, the action decoder is an integral part of the evaluated system, and the 'unlabeled' claim applies only to the world-model training stages, not to the full behavior-learning pipeline. Please quantify the action-labeled data used for Da and clarify this distinction in the abstract and Section 3.4.

minor comments (5)

[Page 1 header] The header reads 'Proceedings of the 42 st International Conference on Machine Learning'; '42 st' should be '42nd'.
[Appendix E.3.4, Figure 20 caption] The caption contains 'a a single action prototype'; please remove the duplicate 'a'.
[Section 3.2.2] The sentence 'Below we explain the process for computing latent actions using InvDynS, which follows a similar procedure to that of InvDynM' is awkward; it seems to mean the opposite, since InvDynS is the single-agent variant. Please rephrase for clarity.
[Table 1 caption] The caption says 'Best two results are highlighted in boldface and underlined, respectively,' but the table formatting is not described clearly in the text; please explain the convention explicitly or ensure the visual highlighting is unambiguous in the PDF.
[Abstract / Section 4.4] The abstract mentions 'Videos and code are available,' but the text provides no direct code repository link; please include the URL of the code, not only the project website.

Circularity Check

1 steps flagged · score 4.0 of 10

One disclosed but real reduction: video-prediction numbers are conditional on latent actions inferred from the ground-truth future; the behavior-learning and world-model claims are otherwise not circular.

fitted input called prediction [Section 4.2 (Table 1); Eqs. (5), (8), (12)]
"For CADDY, PlaySlot and SVG, we predict future frames conditioned on latent actions or vectors inferred from the ground truth sequence. ... ˆzt = InvDyn(St, St+1) ... ˆSt+1 = cOCVP(fS(S1:t) + fp(p1:t) + fv(v1:t))"

At evaluation, the conditioning variable for a transition is computed from that same transition: InvDyn consumes St and St+1 (Eq. 5) to produce the latent action that cOCVP then uses to output St+1 (Eq. 8). Since InvDyn and cOCVP are trained jointly with LSlot = ||Ŝt − ESAVi(Xt)|| (Eq. 12), the Table 1 PSNR/SSIM/LPIPS numbers measure how much of the target future transition survives a low-dimensional, vector-quantized bottleneck, not whether the model forecasts from the past alone. The paper is transparent about this protocol and applies it equally to SVG and CADDY, so the comparisons are fair; still, the advertised 'video prediction' result is a conditional-reconstruction result rather than an open-loop forecast.

full rationale

The only step that reduces by construction is the quantitative video-prediction evaluation, which feeds latent actions inferred from the ground-truth future into the predictor. This is disclosed, is applied to all controllable baselines, and does not affect the other contributions. The behavior-learning experiments are not circular: the policy is trained on latent actions, the action decoder Da is a separately supervised module (Eq. 15) whose small labeled set is unspecified, and the simulator success rates are external outcomes rather than quantities Da is fit to reproduce; the unspecified size of Da's labeled set is a supervision-disclosure/scope concern, not a circular derivation. Self-citations (SOLD expert policy for demonstration collection, OCVP baseline) are data sources and baselines, not load-bearing proofs, so they do not raise the score further.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

PlaySlot's performance depends on hand-chosen hyperparameters (slot counts, action prototype counts, latent dimensions, loss weights) and on two domain assumptions: that SA Vi slots remain consistent across frames, and that latent actions learned from random-exploration data transfer to expert behaviors. No new physical entities are postulated; the hybrid latent action space is an internal model construct whose only evidence is the paper's own action decoder and simulator rollouts.

free parameters (5)

Number of slots per dataset = 8 (BlockPush, Sketchy), 4 (ButtonPress), 3 (GridShapes)
Chosen per dataset; slot count is a free design choice that affects scene decomposition.
Number of action prototypes = 8 (ButtonPress, BlockPush, Sketchy), 5 (GridShapes)
Selected by ablation in Table 4; affects expressivity and control of the latent action space.
Latent action dimension = 16 (ButtonPress, BlockPush, Sketchy), 8 (GridShapes)
Sets the information bottleneck between future conditioning and prediction.
Loss weights = lambda_Img=1, lambda_Slot=1, lambda_VQ=0.25
Hand-tuned to balance reconstruction, slot alignment, and VQ regularization.
Size of action-labeled set for Da = not reported
Action decoder training data size is unspecified but is required for the behavior learning evaluation.

assumptions (4)

domain assumption SA Vi slot decomposition yields temporally consistent, object-bound slots
Appendix A states SA Vi 'fails to generalize to complex real-world robotic scenarios', so the central claims are bounded to simple scenes; if slots drift or split, InvDyn and cOCVP degrade.
domain assumption Latent actions trained on random-exploration videos transfer to expert-demonstration distributions
Table 2 shows PlaySlot drops on expert demos versus random exploration and that continuous latent actions outperform the hybrid variant there, indicating partial transfer only.
domain assumption The action decoder Da can be trained on a small labeled set to map latent actions to executable actions
Behavior learning success is measured by executing decoded actions in the simulator, so the quality of Da is load-bearing; its training set size is not reported.
standard math Standard deep learning machinery (slot attention, transformers, VQ, backpropagation) behaves as intended
Used throughout; standard background results are assumed without proof.

invented entities (1)

Hybrid latent action space (discrete action prototypes plus continuous variability)
purpose: Serves as the control signal for video prediction and as the target for behavior cloning
It is a model-internal latent variable. The paper's action decoder gives an internal consistency check, but no external measurement of the latent space exists.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning." pith.science (2026). https://pith.science/paper/TNY7MZ5E

@misc{pith2026250207600,
  author       = {Pith},
  title        = {Pith review of: PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNY7MZ5E}},
  note         = {Machine review of arXiv:2502.07600}
}

read the original abstract

Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on videos and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows the generation of multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. Videos and code are available on https://play-slot.github.io/PlaySlot/.

Figures

Figures reproduced from arXiv: 2502.07600 by the authors.

**Figure 1.** PlaySlot parses an image X1 into its object components S1. It then predicts multiple future object states and frames with an object-centric video prediction module (cOCVP) conditioned on latent actions Z, which can be inferred from a reference video with our InvDyn module, provided as input, or generated by a learned action policy. Humans naturally achieve such understanding by parsing their environment into a back… view at source ↗

**Figure 2.** Overview of PlaySlot training and inference processes. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Qualitative comparison on (a) ButtonPress and (b) BlockPush datasets. PlaySlot accurately predicts the scene [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: Quantitative results on the GridShapes dataset [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 6.** Figure 6: Qualitative results on a real-world robotics se [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 8.** Figure 8: Predicted frames using latent actions generated [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

**Figure 9.** Figure 9: Success rates (%) of policies learned with PlaySlot under two evaluation protocols on the a) ButtonPress and [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Success rate (%) as a function of available expert demonstrations on the a) ButtonPress and b) BlockPush envi [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Qualitative comparison on (a) ButtonPress and (b) BlockPush datasets. Our method accurately predicts the [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

**Figure 12.** Figure 12: Predicted frames using latent actions from the learned policy, and sequences simulated by executing the decoded [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

**Figure 13.** Figure 13: Predicted frames using latent actions from the learned policy, and simulation computed by executing the decoded [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

**Figure 14.** Figure 14: Qualitative comparison on BlockPush. Our method accurately predicts the scene dynamics and object interac [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]

**Figure 15.** Figure 15: PlaySlot predictions and object representations on a BlockPush sequence. We visualize the ground truth se [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]

**Figure 16.** Figure 16: PlaySlot predictions conditioned on different latent actions, including the inferred inverse dynamics, as well as [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

**Figure 17.** Figure 17: Qualitative comparison on ButtonPress. All methods successfully reconstruct the ground truth sequence by [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]

**Figure 18.** Figure 18: PlaySlot predictions and object representations on a ButtonPress sequence. We visualize the ground truth [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]

**Figure 19.** Figure 19: PlaySlot predictions conditioned on different latent actions, including the inferred inverse dynamics, as well as [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]

**Figure 20.** Figure 20: PlaySlot predictions conditioned on different latent actions, including the inferred inverse dynamics, as well [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]

**Figure 21.** Figure 21: PlaySlot predictions and object representations on a Sketchy sequence. We visualize the ground truth sequence, [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

[3]

On the binding problem in artificial neural networks

Greff, K., Van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208,

arXiv 2012
[5]

Object-centric world model for language-guided manipulation

Jeong, Y ., Chun, J., Cha, S., and Kim, T. Object-centric world model for language-guided manipulation. arXiv preprint arXiv:2503.06170,

arXiv
[7]

Illiterate dall-e learns to compose

Singh, G., Deng, F., and Ahn, S. Illiterate dall-e learns to compose. arXiv preprint arXiv:2110.11405,

arXiv
[8]

Object- centric image to video generation with language guid- ance

Villar-Corrales, A., Plepi, G., and Behnke, S. Object- centric image to video generation with language guid- ance. arXiv preprint arXiv:2502.11655,

work page arXiv
[9]

or SOLV (Aydemir et al., 2023), as well as scale our inverse dynamics and predictor models. Furthermore, we can employ factorized latent action vectors, which represent in a disentangled manner different actions that happen simultaneously, such as moving the robot arm and opening the gripper. We believe that this architectural modifications will enable us...

work page 2023
[12]

The projected object slots are then conditioned by adding them with the projected action prototype and variability embedding from the corresponding time step

To enable predictions conditioned on the inferred latent actions, cOCVP maps the action prototypes p1:t, variability em- beddings v1:t and object slots S1:t into the token dimensionality. The projected object slots are then conditioned by adding them with the projected action prototype and variability embedding from the corresponding time step. Furthermor...

work page 2023
[14]

CADDY infers latent actions that encode the agent’s actions between consecutive pairs of frames

is a recurrent encoder-decoder model designed for playable video generation, enabling user-controllable future video prediction. CADDY infers latent actions that encode the agent’s actions between consecutive pairs of frames. These latent actions are parameterized with a discreteone-hot action label, which determines the high-level action taking place; an...

work page 2023
[15]

In contrast to the object-centric representations employed by PlaySlot, LAPO relies on feature maps output by a convolutional encoder

is a recent model proposed for learning a world model, an inverse dynamics model, and a latent action policy from unlabeled videos. In contrast to the object-centric representations employed by PlaySlot, LAPO relies on feature maps output by a convolutional encoder. ButtonPress Behavior In Fig. 10a), we compare policies learned by PlaySlot and LAPO on the...

work page 2023

Show all 15 references

[1024]

This module aggregates information from the object slots into the [ACT] token, and outputs a single latent action ˆ zt that captures the agent’s action, making it well-suited for single-agent environments. InvDynM: InvDynM independently processes each object slot with a shared...

2017
[2017]

and are trained on a single NVIDIA A100 GPU. B.1. Object-Centric Learning We closely follow Kipf et al. (2022) for the implementation of the SA Vi object-centric decomposition model, which we employ as scene parsing and object rendering modules. We strictly adhere to the archi...

2022
[2018]

To ensure a fair comparison, we balance the number of learnable parameters and compute requirements for all methods

and CADDY (Menapace et al., 2021), as well as the object-centric video prediction models SlotFormer (Wu et al., 2023a) and OCVP (Villar-Corrales et al., 2023). To ensure a fair comparison, we balance the number of learnable parameters and compute requirements for all methods. ...

2021
[2020]

Mastering diverse domains through world models

Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104,

arXiv
[2022]

Object-centric temporal consistency via condi- tional autoregressive inductive biases

Meo, C., Nakano, A., Lic ˘a, M., Didolkar, A., Suzuki, M., Goyal, A., Zhang, M., Dauwels, J., Matsuo, Y ., and Ben- gio, Y . Object-centric temporal consistency via condi- tional autoregressive inductive biases. arXiv preprint arXiv:2410.15728,

arXiv
[2023]

Object discovery from motion-guided to- kens

Bao, Z., Tokmakov, P., Wang, Y .-X., Gaidon, A., and Hebert, M. Object discovery from motion-guided to- kens. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023),

2023
[2024]

P., Matthey, L., Watters, N., Kabra, R., Hig- gins, I., Botvinick, M., and Lerchner, A

Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Hig- gins, I., Botvinick, M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. arXiv:1901.11390,

1901 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [3]

On the binding problem in artificial neural networks

Greff, K., Van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208,

arXiv 2012

[2] [5]

Object-centric world model for language-guided manipulation

Jeong, Y ., Chun, J., Cha, S., and Kim, T. Object-centric world model for language-guided manipulation. arXiv preprint arXiv:2503.06170,

arXiv

[3] [7]

Illiterate dall-e learns to compose

Singh, G., Deng, F., and Ahn, S. Illiterate dall-e learns to compose. arXiv preprint arXiv:2110.11405,

arXiv

[4] [8]

Object- centric image to video generation with language guid- ance

Villar-Corrales, A., Plepi, G., and Behnke, S. Object- centric image to video generation with language guid- ance. arXiv preprint arXiv:2502.11655,

work page arXiv

[5] [9]

or SOLV (Aydemir et al., 2023), as well as scale our inverse dynamics and predictor models. Furthermore, we can employ factorized latent action vectors, which represent in a disentangled manner different actions that happen simultaneously, such as moving the robot arm and opening the gripper. We believe that this architectural modifications will enable us...

work page 2023

[6] [12]

The projected object slots are then conditioned by adding them with the projected action prototype and variability embedding from the corresponding time step

To enable predictions conditioned on the inferred latent actions, cOCVP maps the action prototypes p1:t, variability em- beddings v1:t and object slots S1:t into the token dimensionality. The projected object slots are then conditioned by adding them with the projected action prototype and variability embedding from the corresponding time step. Furthermor...

work page 2023

[7] [14]

CADDY infers latent actions that encode the agent’s actions between consecutive pairs of frames

is a recurrent encoder-decoder model designed for playable video generation, enabling user-controllable future video prediction. CADDY infers latent actions that encode the agent’s actions between consecutive pairs of frames. These latent actions are parameterized with a discreteone-hot action label, which determines the high-level action taking place; an...

work page 2023

[8] [15]

In contrast to the object-centric representations employed by PlaySlot, LAPO relies on feature maps output by a convolutional encoder

is a recent model proposed for learning a world model, an inverse dynamics model, and a latent action policy from unlabeled videos. In contrast to the object-centric representations employed by PlaySlot, LAPO relies on feature maps output by a convolutional encoder. ButtonPress Behavior In Fig. 10a), we compare policies learned by PlaySlot and LAPO on the...

work page 2023