Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

Flow Equivariant World Models claim that encoding self-motion and external object motion as one-parameter flows in a recurrent latent memory lets a world model track out-of-view dynamics and extrapolate far beyond the training horizon.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:54 UTC pith:TKSQPZZF

load-bearing objection A solid, honest paper that unifies self-motion and external object motion in a flow-equivariant memory and shows real empirical gains, but the theory covers only the fully observed case and the velocity channels are hand-matched to the benchmarks' ground-truth velocities. the 3 major comments →

arxiv 2601.01075 v2 pith:TKSQPZZF submitted 2026-01-03 cs.LG cs.AIcs.CV

Flow Equivariant World Models: Memory for Partially Observed Dynamic Environments

Hansen Jin Lillemark , Benhao Huang , Fangneng Zhan , Yilun Du , Thomas Anderson Keller This is my paper
classification cs.LG cs.AIcs.CV
keywords world modelsflow equivariancepartial observabilitylatent memoryvideo predictionequivariant neural networksout-of-view dynamicslength generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a world model's memory should be organized around motion itself: both the agent's own movement and the motion of external objects are treated as one-parameter flows that shift a spatially structured latent memory, rather than being rediscovered frame by frame. Because the memory moves equivariantly with these flows, information about regions outside the agent's field of view keeps flowing correctly while unobserved, so the model can track objects after they leave view and predict far beyond its training horizon. On 2D and 3D partial-observability benchmarks, the resulting Flow Equivariant World Model maintains low error and stable rollouts over hundreds of frames, while diffusion and state-space baselines hallucinate or forget. The authors take this as evidence that predictive representations become more powerful when organized in line with the temporal and dynamical structure of the world.

Core claim

The central claim is that both self-motion and external object motion can be unified as time-parameterized Lie group flows, and that implementing group equivariance with respect to these flows in the latent memory yields a world model that tracks out-of-view dynamics. The paper introduces a generalized flow equivariant recurrence in which each object velocity gets its own memory 'velocity channel' that flows under its own vector field, while the whole latent map is transformed by the inverse of the agent's action to keep it in the agent's egocentric frame. The authors prove by induction that, under a trivial-lift condition on the encoder and equivariance conditions on the encoder and update,

What carries the argument

The load-bearing object is the generalized flow equivariant recurrence relation h_{t+1}(ν) = T^{-1}_{a_t} ψ_1(ν) U_θ[h_t(ν); E_θ[f_t,h_t](ν)]. Here h_t is a latent map with one channel per discrete velocity ν; ψ_1(ν) is a one-step flow (spatial shift or rotation) applied to that channel; T^{-1}_{a_t} is the inverse action transform that keeps the map in the agent's egocentric frame; E_θ is a trivial-lift encoder that duplicates the observation to all velocity channels; and U_θ is an equivariant update, implemented as a convolution or a gated transformer operation. This recurrence makes the hidden state flow with both self-motion and inferred external motion, so unobserved regions of the late

Load-bearing premise

The framework assumes the set of external object velocities is known in advance and can be covered by a small discrete set of flow channels; in both benchmarks V was hand-chosen to match the simulator's ground-truth velocities, so if an unobserved object moves at a velocity not in V, its latent memory channel cannot flow with it and out-of-view tracking fails.

What would settle it

Train or evaluate on a world where an object moves at a velocity not in the discrete channel set V (for example, 0.5 pixel per step in the MNIST World, where V is set to integers from -2 to 2), or where velocities change continuously through acceleration. If FloWM still tracks the object while it is out of view, the claim survives; if the object blurs, teleports, or is forgotten, the reported out-of-view tracking is an artifact of matching V to the simulator's velocities.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A world model can keep representing objects that have left the field of view, because their state lives in a latent map that continues to flow under the stored velocity.
  • Long-horizon prediction no longer requires holding all past frames in a context window; the recurrent map itself is the memory, so sliding-window eviction does not destroy consistency.
  • Group closure means returning to an old viewpoint restores the same latent state, which should eliminate drift and hallucination in loop-closure situations.
  • Flow equivariance gives large training-efficiency gains, suggesting that building motion symmetry into the architecture is a cheap substitute for extra data.
  • The framework is stated for any Lie-group action space and can in principle extend to full 3D neural maps, not just 2D translations and rotations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the framework's viability depends on knowing the set of possible external velocities in advance; a natural stress test is to run an object whose velocity is not in the discrete channel set and measure whether out-of-view tracking degrades.
  • Editorial extension: the group-structured latent map could serve as the world-state representation for planning-oriented, non-generative world-model objectives, not just for video prediction.
  • Editorial extension: the velocity-channel organization resembles velocity-tuned spatial cells in biological navigation; one could probe whether the learned latent channels become selectively responsive to particular directions and speeds.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Flow Equivariant World Models (FloWM), recurrent latent world models whose memory is a set of velocity channels h_t(ν), each flowing under a known one-parameter group action and transformed by the inverse of the agent's action to counteract self-motion. The authors propose a generalized flow-equivariant recurrence (Eq. 5) and provide an inductive proof in Appendix A under exact-equivariance assumptions. Two instantiations are evaluated: a simple convolutional model on 2D MNIST World and a ViT-based model with a latent top-down map on 3D Dynamic Block World. They report that FloWM substantially outperforms diffusion-based baselines (DFoT and DFoT-SSM) and their ablations, retaining low MSE and high SSIM for 150–210 step rollouts, well beyond the training horizon, while also providing a static/textured Block World ablation.

Significance. If the results hold, this is a valuable and timely contribution: it demonstrates that imposing temporal/flow symmetry on a recurrent latent memory can provide long-horizon consistency in partially observed environments, an increasingly central challenge in world modeling. The benchmark design is diagnostic and the final claims are falsifiable. The paper is generally clear, includes detailed hyperparameters and configuration tables, and links to code and datasets, which strengthens reproducibility. The main caveats are that the central mechanism is validated only when the discrete velocity set V exactly matches the simulator's ground-truth velocities, and that the 3D instantiation relies on an unquantified approximate-equivariance assumption that is not covered by the Appendix proof. These issues are load-bearing for the advertised claims but appear addressable with additional experiments and analysis.

major comments (3)
  1. [§3.1 Eq. (5); Appendices F.2/E.2; §6] The central out-of-view tracking property requires the object velocity to lie in the pre-specified discrete set V. Eq. (5) flows only channels ν∈V; an object moving with velocity u∉V has no co-moving channel, so its latent trace drifts across velocity channels and is destroyed by the max-pool readout. In both benchmarks V is hand-chosen to equal the ground-truth simulator velocities: MNIST V={-2..2}^2 (Appendix F.2) and Block World V={±1, no diagonals} with zero (Appendix E.2). Thus the experiments demonstrate out-of-view tracking only for velocities that are known a priori and discrete. Section 6 mentions continuous vs. discrete velocities but not the stronger requirement that V must cover the true velocities. Please add experiments with velocities outside V (e.g., unseen or continuous velocities) or provide a quantitative characterization of degradation when coverage fails.
  2. [§3.2 Transformer-Based FloWM; Appendix A] Appendix A proves flow equivariance only under the assumptions of Eq. (6), i.e., exact group equivariance of the encoder and update. The 3D ViT encoder mapping first-person 3D observations to a top-down map is not analytically equivariant; the paper states "we simply treat the output of the encoder as if it were equivariant" and appeals to prior work for learning approximate equivariance. The claim "in practice this appears to hold" is supported only by aggregate rollout metrics, which conflate approximate equivariance with other sources of performance. Because the formal guarantee of Eq. (11) does not apply to this model, the 3D empirical results rest on an unmeasured assumption. I ask for a direct measurement of equivariance error over rollouts, or a comparison with an exactly equivariant encoder/ablation, before the 3D claims can be accepted.
  3. [§3.1 Eq. (7) and Appendix A] Eq. (7) augments the generalized recurrence with the self-motion transform T^{-1}_{a_t}, but the Appendix proof treats only external-flow equivariance and explicitly states that proofs are maintained in fully observed settings. The partial-observability components — windowed write-in, zero padding, and windowed read-out — are not part of the formal theorem. Since self-motion equivariance is as important as external flow equivariance for the paper's memory claims, especially when rotations are involved in 3D, the manuscript should state the precise assumptions under which Eq. (7) is equivariant, or explicitly separate the theoretical claim from the empirically validated approximation. Without this, the theory section supports a cleaner special case than the architecture actually evaluated.
minor comments (5)
  1. [§3.2, Fig. 4] The text says the latent map is "always egocentric, shifting and rotating around the agent in the center," while later related-work paragraphs describe the model as a formalization of allocentric maps. Please clarify whether the map is egocentric or allocentric and how this distinction is handled in the action transform.
  2. [Appendix G.5] The VAE reconstruction error (about 0.02 MSE for MNIST World) is an irreducible floor for DFoT/DFoT-SSM comparisons. It would strengthen the report to also show pixel-space values or a reconstruction-corrected metric so the reader can separate diffusion error from VAE error.
  3. [Appendix G.4] The statement that 50 vs. 70 context frames "produce similar results" is not accompanied by the supporting numbers. Since this choice affects the fairness of the comparison in Table 2, please report the actual values or remove the claim.
  4. [§4.2, Figure 5(b)] The MSE-vs-step plot would benefit from a log-scale y-axis and error bars/confidence intervals; the current linear scale hides the early differences among the top-performing models.
  5. [Appendix C.2] Table 4 caption notes the numbers are "not easily comparable" across dataset splits because of different statistics. Please state the normalization used for the textured split, or provide a common normalization (e.g., an all-black baseline) for direct comparability.

Circularity Check

0 steps flagged

No circular derivation: predictions are empirically tested against external baselines, the flow-equivariance framework is re-proven in the appendix, and the velocity-channel set is a disclosed dataset prior rather than a fitted parameter repackaged as a prediction.

full rationale

The paper's central mathematical object, flow-equivariant recurrence, is introduced via a self-citation to Keller (2025), a prior work by one of the current authors. However, this is not a circular reduction of the paper's target claim: the generalized recurrence (Eq. 5) is stated and then proved in Appendix A under explicit conditions (trivial lift, equivariant encoder/update, invariant initialization), rather than being assumed from the cited work. The proof follows Keller's technique, but the argument is contained in the paper itself. The empirical claims (out-of-view tracking, long-horizon extrapolation) are evaluated against external diffusion and state-space baselines and against ablations, and no quantity fitted from the target rollouts is renamed as a prediction. The choice of velocity channels V equal to the simulator's ground-truth velocities (MNIST V = {-2..2}^2, Block World V = {±1, no diagonals}) is a dataset-specific modeling prior, disclosed in Appendices F.2 and E.2; the model still must infer each object's velocity from observations and select the appropriate channel, so the result is not forced by construction. The paper also explicitly acknowledges the discrete-velocity limitation and the non-analytic 3D encoder approximation. These are scope limitations, not circular steps. Overall, the derivation is self-contained apart from notation and framing inherited from Keller (2025); this warrants a low score rather than a finding of circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The framework contributes a method, not new physics. The main free parameters are the hand-chosen velocity-channel sets, which are tuned to the exact object velocities in each simulator. The axioms are the Lie-flow assumption on dynamics, exact action representations, the unproven learned equivariance of the 3D encoder, and the unproven extension from fully observed 2D translation equivariance to partial observability and 3D rotations.

free parameters (3)
  • Velocity channel set V (MNIST World) = V = {(-2,-2)...(2,2)}, 25 channels
    Hand-chosen to match digit velocities (-2..+2) in the dataset; not learned. Encodes the external motion prior the model is designed around (Appendix F.2).
  • Velocity channel set V (Block World) = V = {(-1,0),(0,-1),(0,0),(1,0),(0,1)}, 5 channels
    Chosen to match block velocities in the environment (±1, no diagonals); an environment-informed architectural choice (Appendix E.2).
  • Latent map spatial extent = 32x32 for 3D; world size for 2D
    Set to twice the world size for 3D robustness; hand-chosen and affects capacity, though not the core equivariance mechanism (Appendix E.1).
axioms (5)
  • domain assumption The relevant world dynamics and self-motion are generated by one-parameter Lie group flows, i.e. rigid translations/rotations with known parameters.
    This is the core premise of the framework, introduced in the abstract and Section 3.1. It excludes non-rigid, semantic, and stochastic dynamics, as acknowledged in Section 6.
  • domain assumption The representation T_a of the agent's action on the latent memory is known exactly.
    Section 3.1, Eq. 7: 'We assert the representation of the action on this hidden state is known, denoted T_at.' In Block World this holds only because actions are exact grid-aligned translations or 90-degree rotations.
  • ad hoc to paper The non-equivariant 3D encoder can be treated as if equivariant, and the recurrence will cause it to learn approximate equivariance.
    Section 3.2: 'we simply treat the output of the encoder as if it were equivariant in the recurrence relation, and anticipate... the encoder to learn to become equivariant.' No formal argument is given; this is acknowledged in Section 6.
  • domain assumption The fully observed, 2D-translation flow equivariance proof extends to the partially observed, 3D world modeling setting.
    Appendix A proves equivariance only for fully observed sequences with G the 2D translation group, and explicitly says 'we therefore maintain this in our proofs.' The paper does not prove the partial-observation form or the self-action term.
  • domain assumption Dynamics are deterministic given the action sequence; stochastic dynamics are not modeled.
    All experiments use deterministic simulators; Section 6 states that modeling stochastic trajectories is left to future work.
invented entities (1)
  • Velocity-channel latent memory h_t(ν) no independent evidence
    purpose: Represents external object motion by allocating one hidden state per velocity and flowing each channel; combined with the self-action transform it provides the world model's memory.
    This is an architectural construct introduced in Keller (2025) and reused here; no external falsifiable handle is provided beyond the paper's own benchmarks.

pith-pipeline@v1.3.0-alltime-deepseek · 29393 in / 17464 out tokens · 152151 ms · 2026-08-03T12:54:20.853517+00:00 · methodology

0 comments
read the original abstract

Embodied systems experience the world as 'a symphony of flows': a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks. More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/

Figures

Figures reproduced from arXiv: 2601.01075 by Benhao Huang, Fangneng Zhan, Hansen Jin Lillemark, Thomas Anderson Keller, Yilun Du.

Figure 1
Figure 1. Figure 1: Partially observable dynamic world modeling. The agent observes dynamics, turns away, then turns back to the original viewpoint. Flow Equivariant World Models (FloWM) can successfully integrate dynamics through time in a stable manner, while existing work hallucinates. manner. Specifically, we find that both internal and external motion can be understood as math￾ematical ‘flows’, enabling both sources of v… view at source ↗
Figure 2
Figure 2. Figure 2: Comparing World Modeling Frameworks. a) Standard autoregressive video diffusion evicts frames beyond the sliding window. b) Information dependencies between past observations and generated frames cause inconsistency without memory. c) Existing memory solutions are view￾dependent, and thus cannot predict dynamic scenes consistently. d) FloWM remembers past obser￾vations in the spatial latent memory, and con… view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of the Simple Recur￾rent FloWM on MNIST World. FloWM Re￾currence relation. Velocity channels are plotted as rows, with the ‘read-in’ and ‘read-out’ part of the hidden state in blue. In the following subsection, we will propose that in order to gain the efficiency and robustness ben￾efits of equivariance in the world modeling setting, the ‘hidden state’ or memory of a world model can be group-… view at source ↗
Figure 4
Figure 4. Figure 4: Transformer-Based FloWM. a) Image observation ft at time t and FoV selected map latents ht are passed through ViT encoder Eθ. Latent map ht is fully learned, visualized as a map here for clarity. b) Write to ht at the FoV locations, then transform latent map according to known action at and internal flow ψ1(ν), producing ht+1. c) Decode using cross attention over FoV of ht+1 with a ViT decoder Dθ to predic… view at source ↗
Figure 5
Figure 5. Figure 5: Dynamic MNIST World Prediction Rollouts a) Timesteps 0 to 49 are given as obser￾vations. Models are trained to predict up to t = 69. Note that FloWM does not diverge even at timestep 199, while baselines slowly degrade in image quality or lose track of the digits. b) MSE over different length rollouts show length generalization. c) Learning efficiency of the FloWM [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean model predic￾tion error per timestep on 3D Dy￾namic Block World [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Dynamic Block World Prediction Visualizations. Timesteps 0 to 49 are given as ob￾servations for this visualization. Note that FloWM stays consistent until the final frame, while the baselines hallucinate object position and color. The ablations are unable to remain consistent. involving a static version of Block World are included in Appendix C, and more training details are available in Appendix E. Result… view at source ↗
Figure 8
Figure 8. Figure 8: Partially observable dynamic world modeling in 2d environments. Grayed out areas are not visible to the agent at time t. The agent moves its view each timestep via action at and must predict future states after an observation phase, conditioned on its own future action sequence. ants that model time varying geometry or appearance) further support rerendering observed motions from novel viewpoints (Pumarola… view at source ↗
Figure 9
Figure 9. Figure 9: Textured 3D Block World Rollouts. Here we visualize additional qualitative rollout results on the Textured 3D Block World split. Note the hallucinated objects in both DFoT and DFoT-SSM as time goes on, whereas FloWM remains consistent. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Rollout Error (MSE) vs. Forward Prediction Steps for all data subsets of MNIST World. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 conditional novelty 7.0

    Current video world models do not reliably recover an object's updated state after it disappears and reappears under simultaneous camera and scene dynamics.

  2. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 7.0

    MemoBench curates 360 ground-truth clips and an evaluation suite to diagnose memory consistency failures in video models when objects change state while out of view.

  3. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 7.0

    MemoBench is a new diagnostic benchmark with 360 synthetic and real clips plus VQA evaluation that tests memory consistency in video models under the disappear-and-reappear paradigm in dynamically changing environments.

  4. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 conditional novelty 7.0

    None of ten tested video-generation models reliably remembers objects after occlusion in dynamic scenes; static-camera videos inflate consistency scores.

  5. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 7.0

    MemoBench is a new diagnostic benchmark with automated and VQA metrics that evaluates memory consistency in video models under disappear-and-reappear in dynamic environments.

  6. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 6.0

    MemoBench curates 360 clips and an evaluation suite to test video models on recovering updated object states after disappear-and-reappear in changing environments.

  7. Echo-Memory: A Controlled Study of Memory in Action World Models

    cs.CV 2026-06 unverdicted novelty 6.0

    A controlled study finds that block-wise state-space recurrence outperforms other memory designs for open-domain scene return in action-conditioned video models, and that standard replay metrics do not adequately meas...

Reference graph

Works this paper leans on

67 extracted references · 44 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Cosmos world foundation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xia...

    Pith/arXiv arXiv 2025

  3. [3]

    a ron van den Oord, Inbar Mosseri, Adrian Bolton, Satinder Singh, and Tim Rockt \

    Philip J. Ball, Jakob Bauer, Frank Belletti, Bethanie Brownfield, Ariel Ephrat, Shlomi Fruchter, Agrim Gupta, Kristian Holsheimer, Aleksander Holynski, Jiri Hron, Christos Kaplanis, Marjorie Limont, Matt McGill, Yanko Oliveira, Jack Parker-Holder, Frank Perbet, Guy Scully, Jeremy Shar, Stephen Spencer, Omer Tov, Ruben Villegas, Emma Wang, Jessica Yung, Ci...

    2025

  4. [4]

    Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E

    Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13 0 (1): 0 2453, 2022. doi:10.1038/s41467-022-29939-5. URL https://doi.org/10.1038/s414...

    doi:10.1038/s41467-022-29939-5 2022

  5. [5]

    Egomap: Projective mapping and structured egocentric memory for deep rl, 2020

    Edward Beeching, Christian Wolf, Jilles Dibangoye, and Olivier Simonin. Egomap: Projective mapping and structured egocentric memory for deep rl, 2020. URL https://arxiv.org/abs/2002.02286

    Pith/arXiv arXiv 2020

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/video-generation-models-as-world-simulators

    2024

  7. [7]

    Stanley H. Chan. Tutorial on diffusion models for imaging and vision, 2025. URL https://arxiv.org/abs/2403.18103

    Pith/arXiv arXiv 2025

  8. [8]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion, 2024

    Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion, 2024. URL https://arxiv.org/abs/2407.01392

    Pith/arXiv arXiv 2024

  9. [9]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks, 2023

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks, 2023. URL https://arxiv.org/abs/2306.13831

    Pith/arXiv arXiv 2023

  10. [10]

    Group equivariant convolutional networks

    Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp.\ 2990--2999, New York, New York, USA, 20--22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/co...

    2016

  11. [11]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Oasis: A universe in a transformer

    Etched Decart, Quinn McIntyre, Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe in a transformer. URL: https://oasis-model. github. io, 2024

    2024

  13. [13]

    Latham, Timothy E

    William Dorrell, Peter E. Latham, Timothy E. J. Behrens, and James C. R. Whittington. Actionable neural representations: Grid cells from minimal constraints, 2023. URL https://arxiv.org/abs/2209.15563

    Pith/arXiv arXiv 2023

  14. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929

    Pith/arXiv arXiv 2021

  15. [15]

    The matrix: Infinite-horizon world generation with real-time moving control, 2024

    Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real-time moving control, 2024. URL https://arxiv.org/abs/2412.03568

    Pith/arXiv arXiv 2024

  16. [16]

    seq-jepa: Autoregressive predictive learning of invariant-equivariant world models, 2025

    Hafez Ghaemi, Eilif Muller, and Shahab Bakhtiari. seq-jepa: Autoregressive predictive learning of invariant-equivariant world models, 2025. URL https://arxiv.org/abs/2505.03176

    arXiv 2025

  17. [17]

    Mineworld: a real-time and open-source interactive world model on minecraft, 2025

    Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft, 2025. URL https://arxiv.org/abs/2504.08388

    Pith/arXiv arXiv 2025

  18. [18]

    Photorealistic video generation with diffusion models, 2023

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023. URL https://arxiv.org/abs/2312.06662

    Pith/arXiv arXiv 2023

  19. [19]

    World models

    David Ha and J \"u rgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2 0 (3), 2018

    Pith/arXiv arXiv 2018

  20. [20]

    Td-mpc2: Scalable, robust world models for continuous control, 2024

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024. URL https://arxiv.org/abs/2310.16828

    Pith/arXiv arXiv 2024

  21. [21]

    Masked autoencoders are scalable vision learners, 2021

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. URL https://arxiv.org/abs/2111.06377

    Pith/arXiv arXiv 2021

  22. [22]

    Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

    Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, et al. Matrix-game 2.0: An open-source, real-time, and streaming interactive world model. arXiv preprint arXiv:2508.13009, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL https://arxiv.org/abs/2207.12598

    Pith/arXiv arXiv 2022

  24. [24]

    Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion

    Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion. arXiv preprint arXiv:2410.19324, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Self forcing: Bridging the train-test gap in autoregressive video diffusion, 2025

    Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion, 2025. URL https://arxiv.org/abs/2506.08009

    Pith/arXiv arXiv 2025

  26. [26]

    Sensorimotor mismatch signals in primary visual cortex of the behaving mouse

    Georg B Keller, Tobias Bonhoeffer, and Mark H \"u bener. Sensorimotor mismatch signals in primary visual cortex of the behaving mouse. Neuron, 74 0 (5): 0 809--815, 2012

    2012

  27. [27]

    Anderson Keller

    T. Anderson Keller. Flow equivariant recurrent neural networks, 2025. URL https://arxiv.org/abs/2507.14793

    arXiv 2025

  28. [28]

    Anderson Keller and Max Welling

    T. Anderson Keller and Max Welling. Topographic vaes learn equivariant capsules, 2022. URL https://arxiv.org/abs/2109.01394

    Pith/arXiv arXiv 2022

  29. [29]

    3d gaussian splatting for real-time radiance field rendering, 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering, 2023. URL https://arxiv.org/abs/2308.04079

    Pith/arXiv arXiv 2023

  30. [30]

    Grewe, and Bernhard Schölkopf

    Hamza Keurti, Hsiao-Ru Pan, Michel Besserve, Benjamin F. Grewe, and Bernhard Schölkopf. Homomorphism autoencoder -- learning group structured representations from observed transitions, 2024. URL https://arxiv.org/abs/2207.12067

    Pith/arXiv arXiv 2024

  31. [31]

    Kingma and Ruiqi Gao

    Diederik P. Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation, 2023. URL https://arxiv.org/abs/2303.00848

    Pith/arXiv arXiv 2023

  32. [32]

    Regular SE(3) Group Convolutions for Volumetric Medical Image Analysis, pp.\ 252--261

    Thijs Kuipers and Erik Bekkers. Regular SE(3) Group Convolutions for Volumetric Medical Image Analysis, pp.\ 252--261. 10 2023. ISBN 978-3-031-43897-4. doi:10.1007/978-3-031-43898-1_25

    doi:10.1007/978-3-031-43898-1_25 2023

  33. [33]

    A path towards autonomous machine intelligence version 0.9.2, 2022-06-27

    Yann LeCun and Courant. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. 2022. URL https://api.semanticscholar.org/CorpusID:251881108

    2022

  34. [34]

    A sensorimotor circuit in mouse cortex for visual flow predictions

    Marcus Leinweber, Daniel R Ward, Jan M Sobczak, Alexander Attinger, and Georg B Keller. A sensorimotor circuit in mouse cortex for visual flow predictions. Neuron, 95 0 (6): 0 1420--1432, 2017

    2017

  35. [35]

    Dynamic gaussians mesh: Consistent mesh reconstruction from dynamic scenes, 2025

    Isabella Liu, Hao Su, and Xiaolong Wang. Dynamic gaussians mesh: Consistent mesh reconstruction from dynamic scenes, 2025. URL https://arxiv.org/abs/2404.12379

    Pith/arXiv arXiv 2025

  36. [36]

    Parallelizing linear recurrent neural nets over sequence length, 2018

    Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length, 2018. URL https://arxiv.org/abs/1709.04057

    Pith/arXiv arXiv 2018

  37. [37]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020. URL https://arxiv.org/abs/2003.08934

    Pith/arXiv arXiv 2020

  38. [38]

    Neural map: Structured memory for deep reinforcement learning, 2017

    Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning, 2017. URL https://arxiv.org/abs/1702.08360

    Pith/arXiv arXiv 2017

  39. [39]

    Learning symmetric embeddings for equivariant world models, 2022

    Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models, 2022. URL https://arxiv.org/abs/2204.11371

    Pith/arXiv arXiv 2022

  40. [40]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https://arxiv.org/abs/2212.09748

    Pith/arXiv arXiv 2023

  41. [41]

    Approximation-generalization trade-offs under (approximate) group equivariance, 2025

    Mircea Petrache and Shubhendu Trivedi. Approximation-generalization trade-offs under (approximate) group equivariance, 2025. URL https://arxiv.org/abs/2305.17592

    Pith/arXiv arXiv 2025

  42. [42]

    Long-context state-space video world models, 2025

    Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models, 2025

    2025

  43. [43]

    D-nerf: Neural radiance fields for dynamic scenes, 2020

    Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes, 2020. URL https://arxiv.org/abs/2011.13961

    Pith/arXiv arXiv 2020

  44. [44]

    Equivariance through parameter-sharing

    Siamak Ravanbakhsh, Jeff Schneider, and Barnab \'a s P \'o czos. Equivariance through parameter-sharing. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 2892--2901. PMLR, 06--11 Aug 2017. URL https://proceedings.mlr.press/v70/ravanbak...

    2017

  45. [45]

    High-resolution image synthesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL https://arxiv.org/abs/2112.10752

    Pith/arXiv arXiv 2022

  46. [46]

    Statespacediffuser: Bringing long context to diffusion world models

    Nedko Savov, Naser Kazemi, Deheng Zhang, Danda Pani Paudel, Xi Wang, and Luc Van Gool. Statespacediffuser: Bringing long context to diffusion world models. arXiv preprint arXiv:2505.22246, 2025

    arXiv 2025

  47. [47]

    Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling, 2023. URL https://arxiv.org/abs/2208.04933

    Pith/arXiv arXiv 2023

  48. [48]

    History-guided video diffusion, 2025

    Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion, 2025. URL https://arxiv.org/abs/2502.06764

    Pith/arXiv arXiv 2025

  49. [49]

    Towards accurate generative models of video: A new metric & challenges, 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges, 2019. URL https://arxiv.org/abs/1812.01717

    Pith/arXiv arXiv 2019

  50. [50]

    Worrall, Herke van Hoof, Frans A

    Elise van der Pol, Daniel E. Worrall, Herke van Hoof, Frans A. Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning, 2021. URL https://arxiv.org/abs/2006.16908

    Pith/arXiv arXiv 2021

  51. [51]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  52. [52]

    4d gaussian splatting for real-time dynamic scene rendering, 2024

    Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering, 2024. URL https://arxiv.org/abs/2310.08528

    Pith/arXiv arXiv 2024

  53. [53]

    Video world models with long-term spatial memory, 2025

    Tong Wu, Shuai Yang, Ryan Po, Yinghao Xu, Ziwei Liu, Dahua Lin, and Gordon Wetzstein. Video world models with long-term spatial memory, 2025. URL https://arxiv.org/abs/2506.05284

    Pith/arXiv arXiv 2025

  54. [54]

    Xing, and Zhiting Hu

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Pandora: Towards general world model with natural language actions and video states, 2024. URL https://arxiv.org/abs/2406.09455

    Pith/arXiv arXiv 2024

  55. [55]

    Jiannan Xiang, Yi Gu, Zihan Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Guangyi Liu, Yichi Yang, Kun Zhou, Davit Abrahamyan, Arif Ahmad, Ganesh Bannur, Junrong Chen, Kimi Chen, Mingkai Deng, Ruobing Han, Xinqi Huang, Haoqiang Kang, Zheqi Liu, Enze Ma, Hector Ren, Yashowardhan Shinde, Rohan Shingre, Ramsundar Tanikella, Kaiming Tao, Dequan Yang, Xin...

    arXiv 2025

  56. [56]

    Worldmem: Long-term consistent world simulation with memory, 2025

    Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan. Worldmem: Long-term consistent world simulation with memory, 2025. URL https://arxiv.org/abs/2504.12369

    arXiv 2025

  57. [57]

    Cogvideox: Text-to-video diffusion models with an expert transformer, 2025

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer, 2025. URL https://arxiv.org/abs/2408.06072

    Pith/arXiv arXiv 2025

  58. [58]

    Freeman, Fredo Durand, Eli Shechtman, and Xun Huang

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models, 2025. URL https://arxiv.org/abs/2412.07772

    arXiv 2025

  59. [59]

    Frame context packing and drift prevention in next-frame-prediction video diffusion models, 2025 a

    Lvmin Zhang, Shengqu Cai, Muyang Li, Gordon Wetzstein, and Maneesh Agrawala. Frame context packing and drift prevention in next-frame-prediction video diffusion models, 2025 a . URL https://arxiv.org/abs/2504.12626

    arXiv 2025

  60. [60]

    Vsa: Faster video diffusion with trainable sparse attention, 2025 b

    Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. Vsa: Faster video diffusion with trainable sparse attention, 2025 b . URL https://arxiv.org/abs/2505.13389

    arXiv 2025

  61. [61]

    Shine-mapping: Large-scale 3d mapping using sparse hierarchical implicit neural representations, 2023

    Xingguang Zhong, Yue Pan, Jens Behley, and Cyrill Stachniss. Shine-mapping: Large-scale 3d mapping using sparse hierarchical implicit neural representations, 2023. URL https://arxiv.org/abs/2210.02299

    Pith/arXiv arXiv 2023

  62. [62]

    Learning 3d persistent embodied world models

    Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Peihao Chen, Dit-Yan Yeung, and Chuang Gan. Learning 3d persistent embodied world models. arXiv preprint arXiv:2505.05495, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    Oswald, and Marc Pollefeys

    Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam, 2022. URL https://arxiv.org/abs/2112.12130

    Pith/arXiv arXiv 2022

  64. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  65. [65]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  66. [66]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  67. [67]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...