Pith. sign in

REVIEW 4 major objections 5 minor 64 references

On the Benefits of Instance Decomposition in Video Prediction Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that separating a video frame into individual objects, predicting each object's future in its own latent slot, and letting slots attend to one another produces better future frames than a similarly sized whole-scene model.

desk verdict A useful framework with striking qualitative results, but the 'similar capacity' central claim is contradicted by the paper's own parameter tables. read the letter →

arxiv 2501.10562 v1 pith:EGK3W72M submitted 2025-01-17 cs.CV

Eliyas Suleyman , Paul Henderson , Nicolas Pugeault This is my paper · ORCID
classification cs.CV
keywords videopredictionobject-centricrepresentationinstancedecompositionlatenttransformercross-attentionVQ-VAEparametersharingobjectinteractions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that a video prediction model does better when it first splits each frame into separate objects, predicts each object's motion in its own latent code, and then lets those codes attend to one another. To make the test fair, it builds three versions of the same latent-transformer pipeline: a monolithic single-slot model, a decomposed model whose object slots never interact, and a decomposed model with cross-attention between slots. On five datasets ranging from a pedestrian walking to colliding three-dimensional objects, the fully decomposed model matches or beats a much larger monolithic model, with the largest gains on scenes containing collisions and many moving objects. The paper concludes that explicit instance decomposition improves prediction quality, and that cross-attention between instances is the mechanism that turns separation into better interaction prediction.

What carries the argument

The central machinery is an object-aware autoencoder (OAAE) paired with a multi-slot transformer. In the OAAE, a panoptic segmentation model produces one non-overlapping binary mask per object, each masked frame is passed through an encoder and vector-quantized codebook specific to its semantic class, and all instance latents are concatenated and decoded by a single joint decoder. The prediction transformer then treats each instance as a slot, applying factored spatial and temporal attention for self-attention within an instance and cross-attention between instances; all slots of the same class share weights. This machinery lets the paper vary exactly one thing at a time: whether the latent space is split into objects, and whether the object slots exchange information. Cross-attention between instance slots is the mechanism that encodes object interactions and is the load-bearing difference between the interacting and non-interacting variants.

What would settle it

Measure the rate at which instance identities swap or flicker in the YOLOv8 masks on the Real-Traffic test clips, and re-run the SCAT-versus-Single-Slot comparison with masks forced to keep stable identities (for example, via a tracker or human-annotated object IDs). If the quality gap shrinks or disappears when identity switches are removed, the reported benefit is partly an artifact of unstable masks rather than a property of instance decomposition.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the discovery is that decomposing a dynamic scene into per-instance masked latents improves future-frame prediction relative to a non-decomposed model of comparable capacity, even when both use the same latent-transformer machinery. The decomposition is implemented through an object-aware autoencoder that encodes each segmented instance with a class-specific encoder and codebook, producing a structured latent vector that can be decoded jointly. The follow-on transformer predicts each instance's future latents with self-attention over its own spatial-temporal history and cross-attention over every other instance; parameters are shared across instances of the same class. On synthetic scenes with strong collisions, the cross-attending decomposed model (SCAT) is the only variant that keeps objects separated and preserves their shape through impacts, while the non-interacting decomposed variant (SNCAT) often fails precisely on collisions. The paper therefore claims that instance decomposition helps, but only fully pays off when object interactions are modeled.

Load-bearing premise

The load-bearing premise is that the per-frame instance masks passed to the model identify the same physical object over time, so each slot always contains one consistent trajectory; the paper gets this for free on synthetic data, but for the real traffic data it relies on per-frame detections and does not describe any tracking or association step that would keep instance identities stable.

Editorial extensions

If this is right

  • Latent-transformer video predictors can be made much smaller (roughly an order of magnitude fewer parameters on the real traffic data) without losing prediction quality, as long as they decompose the scene into objects and share weights within classes.
  • Scenes with strong interactions such as collisions require cross-attention between object slots; separation alone, without interaction modeling, is not enough and can even hurt when objects collide.
  • The gains from decomposition are largest in multi-object scenes with interactions, so object-centric structure matters most where monolithic models blur or merge objects.
  • Because the architecture supports both decomposed and monolithic variants in one framework, future comparisons of video prediction models can isolate the effect of object-centric structure rather than confounding it with model size.
  • Per-class weight sharing means that adding a new object class adds class-specific parameters, not one new parameter set per visible instance, which makes the approach scale to scenes with many objects of a limited number of types.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the supervised segmenter with an unsupervised object-discovery module that is not given stable identity labels; the paper's supervised masks make the cleanest comparison, but real deployment would need to handle drifting instance boundaries.
  • The per-class weight-sharing scheme implies a scaling story the paper does not develop: the cost of modeling more object types should grow with the class vocabulary rather than with the number of visible instances, which could be probed with large synthetic scenes.
  • On real traffic video the method relies on per-frame detections with no described tracking or association step, so the reported gains on that dataset are an upper bound for what a system with noisy identity switching would achieve; adding an association module may be necessary.
  • If decomposition works because it reduces the statistical burden per object, the benefit should widen when training data is scarce; that is a direct consequence of the paper's framing and could be tested with training-curve experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a family of latent-transformer video prediction models that either encode the whole scene (SiS), decompose into instances with per-class weight sharing (SNCAT), or additionally use cross-attention between instances (SCAT). The central claim, stated in the abstract, is that explicit instance decomposition leads to higher-quality predictions than non-decomposed models of similar capacity. Experiments are conducted on KTH, Real-Traffic, CLEVR-2, CLEVR-3, and Kubric-Real, comparing the three variants and external baselines. The paper concludes that decomposition and cross-attention improve prediction quality, especially in scenes with strong interactions.

Significance. If the empirical claims were supported, the paper would offer a systematic, controlled study of object decomposition in modern latent-transformer video prediction, with a unified architecture family and released dataset-generation code. The cross-attention design and per-class weight sharing are reasonable components. However, the central comparison is confounded by a large parameter-count gap between the decomposed and non-decomposed models, and the appendix reveals test-set-based selection of sampling temperatures. These issues undermine the main empirical claim that decomposition alone is beneficial at similar capacity. The framework itself is a useful starting point, but the evidence as presented does not establish the paper's headline conclusion.

major comments (4)
  1. [Section 4, Tables 1-2; Section 3.2; Appendix Table 5] The claim of comparing models of 'similar capacity' is directly contradicted by the paper's own parameter counts: on KTH, Real-Traffic, CLEVR-2, CLEVR-3, and Kubric-Real, SiS has 48M/286M/105M/186M/287M parameters against SCAT's 23M/28M/25M/26M/40M, a gap of roughly 2-10x. The appendix explains that the non-decomposed VQ-VAE embedding dimension is set to N times the per-instance dimension, and Section 3.2 states that SiS 'result[s] in considerably more parameters.' Thus the comparison does not isolate the effect of decomposition; differences in capacity and optimization trajectory are confounded. This invalidates the abstract's central claim as currently supported.
  2. [Appendix, 'More Results'] The appendix states that results are produced by sampling with a set of 10 temperature values from 0.1 to 0.9 and 'then the best result is selected to compare.' Selecting the best temperature per metric on the test set is a form of test-set overfitting and inflates the reported numbers. This affects all internal comparisons and makes the absolute performance claims unreliable. The authors should select temperatures on a validation split or report results across all temperatures.
  3. [Section 4, Real-Traffic paragraph] The quantitative evaluation on Real-Traffic is performed on a subset of test clips selected by proximity: clips are kept only when the distance between any pair of cars is less than 25% of the image size, yielding 807 clips. This is a test-set-based selection that may bias the comparison toward scenarios that favor decomposition (or, conversely, may make the task easier in ways that are not controlled). The paper should report results on the full test set as well, or provide evidence that the subset does not change the relative ranking of models.
  4. [Section 3.1 and Section 4 (Real-Traffic)] For Real-Traffic, instance masks come from per-frame YOLOv8 detections, but the paper does not describe any tracking or association mechanism that links detections across time. Without consistent instance identities, slots may flicker, split, or swap between frames, corrupting the input trajectories to the decomposed model. This is especially relevant because the model assumes each slot corresponds to the same physical object over the conditioning and prediction horizon. The authors should describe the association method or demonstrate that identity switches do not affect the results.
minor comments (5)
  1. [Section 3.1, Eqs. (4)-(5)] The subscript/superscript notation for class and instance indices is inconsistent between the text and equations (e.g., 'z_c^k' versus 'z_k^c'), making the summation ranges ambiguous. Please standardize the notation.
  2. [Section 3, model list] There is a typo in the variant name: 'Stochastic non-Class Attended Transformer (SNCAT)' is spelled 'Attendted' in the definition list. Please correct.
  3. [Table 1 vs Table 3 and text] The reported KTH PSNR for SCAT differs across the paper: Table 1 reports 25.61, Table 3 reports 25.52, and the text in Section 4 reports 25.46. These discrepancies should be reconciled.
  4. [Section 4, Table 3 and surrounding text] The text claims that baseline hyperparameters were adjusted to match the model size, but Table 3 shows that VideoGPT and SimVP have significantly more parameters than SCAT on several datasets (e.g., 67M vs 40M on Kubric-Real). This undermines the comparison narrative; the actual matching criterion should be stated clearly.
  5. [Section 3.2, Eq. (9)] The cross-attention equation uses an unusual 'M' symbol to denote concatenation over instances; standard notation (e.g., a concatenation symbol or direct sum) would be clearer. The text also refers to Figure 2, but the figure is not labeled in the text body, making it hard to locate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical comparison with self-contained experiments, not a derivation that reduces to its inputs.

full rationale

This paper makes an empirical claim: explicitly decomposing scenes into instances improves latent-transformer video prediction relative to a non-decomposed model. There is no derivation chain whose conclusion is equivalent to its premises. The reported results come from training and evaluating model variants on external datasets (KTH, Real-Traffic) and on synthetic datasets generated with Kubric; the metrics (PSNR, SSIM, LPIPS) are computed against ground-truth frames, not derived from the model definitions. The self-citations (Henderson and Lampert 2020; Henderson, Lampert, and Bickel 2021; Shi et al. 2024) appear only in related-work context and are not load-bearing for the architecture or the experimental conclusions. No uniqueness theorem or prior result by the same authors is invoked to make the comparison forced. Two reviewer concerns are real but are not circularity under the stated rules: the appendix reports that SiS uses an N-times larger embedding dimension and the paper itself concedes SiS has 'considerably more parameters', so the headline 'similar capacity' comparison is confounded with parameter count; and the appendix selects the best of 10 sampling temperatures per metric on the test set, which may inflate reported scores. Both are experimental-validity issues about confounds and test-set selection, not instances where a prediction is identical to a fitted input by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the decomposition being faithful (stable masks, fixed slot budget) and on the VQ-VAE codebook being a sufficient representation. The main free parameter affecting the reported numbers is the test-set-selected sampling temperature, plus the hand-chosen Real-Traffic proximity threshold.

free parameters (3)
  • sampling temperature = best of 10 values in [0.1, 0.9], chosen per dataset on the test set
    The appendix says the reported results are produced by sampling with 10 temperatures and 'the best result is selected to compare.' This is a hyperparameter fitted to the test set, and it directly inflates the reported metrics.
  • Real-Traffic proximity threshold = 0.25 of image size
    Test clips are selected for evaluation only when any pair of cars is closer than 25% of the image size. This hand-chosen threshold defines the subset on which the comparison is reported.
  • loss weights alpha and beta = not specified in paper
    Equation 7 defines L_oaae = L_recon + alpha*L_feature + L_VQ + beta*L_commitment. The chosen values are not reported, so the exact training objective is not reproducible.
assumptions (3)
  • domain assumption Instance segmentation masks are stable and identity-preserving across time.
    The whole multi-slot architecture assumes each slot tracks one object. Section 3.1 describes per-frame masking but not tracking or association.
  • domain assumption The number of instances and their class distribution is known at inference time.
    Slots are allocated per class with a fixed total (e.g., 5 for Kubric-Real in Table 5). The paper does not explain how the model handles scenes with a varying number of objects.
  • domain assumption Codebook prediction via cross-entropy is a valid proxy for frame prediction.
    The transformer predicts discrete VQ-VAE codebook indices, and the reported image metrics depend on the OAAE decoding quality. If the autoencoder is poor, even a perfect codebook predictor would yield bad frames.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Benefits of Instance Decomposition in Video Prediction Models." pith.science (2026). https://pith.science/paper/EGK3W72M

@misc{pith2026250110562,
  author       = {Pith},
  title        = {Pith review of: On the Benefits of Instance Decomposition in Video Prediction Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EGK3W72M}},
  note         = {Machine review of arXiv:2501.10562}
}
read the original abstract

Video prediction is a crucial task for intelligent agents such as robots and autonomous vehicles, since it enables them to anticipate and act early on time-critical incidents. State-of-the-art video prediction methods typically model the dynamics of a scene jointly and implicitly, without any explicit decomposition into separate objects. This is challenging and potentially sub-optimal, as every object in a dynamic scene has their own pattern of movement, typically somewhat independent of others. In this paper, we investigate the benefit of explicitly modeling the objects in a dynamic scene separately within the context of latent-transformer video prediction models. We conduct detailed and carefully-controlled experiments on both synthetic and real-world datasets; our results show that decomposing a dynamic scene leads to higher quality predictions compared with models of a similar capacity that lack such decomposition.

Figures

Figures reproduced from arXiv: 2501.10562 by the authors.

Figure 1
Figure 1. Top: Our proposed multi-object interacting model SCAT. First the input frames are decomposed via a segmentation model, then each decomposed sequence passes through class-specific encoder to convert the 2D frames into latent represen￾tations; then class-specific transformer blocks learn and predict the dynamics of each instance and its relationships with other instances in latent space; lastly, the predicted latent r… view at source ↗
Figure 2
Figure 2. Left: Architecture of the multi-object latent trans￾former. Right: Detail of spatial and temporal attention blocks. SiS. This only takes the original non-segmented frame as input. It is processed by a single encoder, with the latent size matched to the total latent size (over all instances) for model SCAT. In terms of losses, Lrecon remains unchanged, LV Q, Lcommitment and Lf eature will be a modified to a single te… view at source ↗
Figure 3
Figure 3. Comparison of different model variants on the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (29 more)
Figure 4
Figure 4. Figure 4: Qualitative results from our full model and baselines on [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Mean LPIPS Values on Real￾Traffic datasets Dataset Details Decomposition For KTH, we use CLIPSeg with the prompt ’person’ and ’background’ to decompose the frames. For Real-Traffic, we use YOLOv8 to be our in￾stance segmentor. For Kubric generated dataset…
Figure 6
Figure 6. Figure 6: Comparison of Mean LPIPS Values on KTH datasets [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Comparison of Mean LPIPS Values on CLEVR-2 dataset 10 15 20 25 0.1 0.2 Time Steps Mean LPIPS SCAT SNCAT SiS [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Comparison of Mean LPIPS Values on CLEVR-3 dataset 10 15 20 25 0.1 0.2 0.3 Time Steps Mean LPIPS SCAT SNCAT SiS [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Mean LPIPS Values on Kubric-Real dataset [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Mean and Std (Standard Deviation) of LPIPS metric on [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Mean and Std (Standard Deviation) of LPIPS metric on [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Mean and Std of LPIPS metric on CLEVR-2 dataset [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Mean and Std of LPIPS metric on CLEVR-3 dataset 10 15 20 25 0.1 0.2 0.3 Time Steps LPIPS±Std SCAT 10 15 20 25 0.1 0.2 0.3 Time Steps LPIPS±Std SNCAT 10 15 20 25 0.1 0.2 0.3 Time Steps LPIPS±Std SiS [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Mean and Std of LPIPS metric on Kubric-Real dataset Input Prediction t = 1 t = 5 t = 7 t = 9 t = 11 t = 13 t = 15 t = 18 t = 25 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Kubric-Real Example 1 [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 16
Figure 16. Figure 16: Kubric-Real Example 2 Input Prediction t = 1 t = 5 t = 7 t = 9 t = 11 t = 13 t = 15 t = 18 t = 25 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Kubric-Real Example 3 [PITH_FULL_IMAGE:figures/full_fig_p015_17.png]
Figure 18
Figure 18. Figure 18: Kubric-Real Example 4 Input Prediction t = 1 t = 5 t = 7 t = 9 t = 11 t = 13 t = 15 t = 18 t = 25 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]
Figure 19
Figure 19. Figure 19: Kubric-Real Example 5 [PITH_FULL_IMAGE:figures/full_fig_p016_19.png]
Figure 20
Figure 20. Figure 20: Real-Traffic Example 1 Input Prediction t = 1 t = 5 t = 6 t = 7 t = 8 t = 9 t = 10 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p017_20.png]
Figure 21
Figure 21. Figure 21: Real-Traffic Example 2 [PITH_FULL_IMAGE:figures/full_fig_p017_21.png]
Figure 22
Figure 22. Figure 22: Real-Traffic Example 3 Input Prediction t = 1 t = 5 t = 6 t = 7 t = 8 t = 9 t = 10 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p018_22.png]
Figure 23
Figure 23. Figure 23: Real-Traffic Example 4 [PITH_FULL_IMAGE:figures/full_fig_p018_23.png]
Figure 24
Figure 24. Figure 24: CLEVR-3 Example 1 Input Prediction t = 1 t = 5 t = 7 t = 9 t = 11 t = 13 t = 15 t = 18 t = 25 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p019_24.png]
Figure 25
Figure 25. Figure 25: CLEVR-3 Example 2 [PITH_FULL_IMAGE:figures/full_fig_p019_25.png]
Figure 26
Figure 26. Figure 26: CLEVR-3 Example 3 Input Prediction t = 1 t = 5 t = 7 t = 9 t = 11 t = 13 t = 15 t = 18 t = 25 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p020_26.png]
Figure 27
Figure 27. Figure 27: CLEVR-3 Example 4 [PITH_FULL_IMAGE:figures/full_fig_p020_27.png]
Figure 28
Figure 28. Figure 28: KTH Example 1 Input Prediction t = 1 t = 5 t = 7 t = 10 t = 13 t = 16 t = 20 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p021_28.png]
Figure 29
Figure 29. Figure 29: KTH Example 2 [PITH_FULL_IMAGE:figures/full_fig_p021_29.png]
Figure 30
Figure 30. Figure 30: KTH Example 3 Input Prediction t = 1 t = 5 t = 7 t = 10 t = 13 t = 16 t = 20 GT SiS SNCAT SCAT [PITH_FULL_IMAGE:figures/full_fig_p022_30.png]
Figure 31
Figure 31. Figure 31: KTH Example 4 [PITH_FULL_IMAGE:figures/full_fig_p022_31.png]
Figure 32
Figure 32. Figure 32: KTH Example 5 [PITH_FULL_IMAGE:figures/full_fig_p023_32.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 37 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bei, X.; Yang, Y.; and Soatto, S. 2021. Learning Semantic-Aware Dynamics for Video Prediction. CoRR, abs/2104.09762

    work page Pith review arXiv 2021

  4. [4]

    Blattmann, A.; Dockhorn, T.; Kulal, S.; Mendelevitch, D.; Kilian, M.; Lorenz, D.; Levi, Y.; English, Z.; Voleti, V.; Letts, A.; et al. 2023 a . Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127

    arXiv 2023

  5. [5]

    W.; Fidler, S.; and Kreis, K

    Blattmann, A.; Rombach, R.; Ling, H.; Dockhorn, T.; Kim, S. W.; Fidler, S.; and Kreis, K. 2023 b . Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. In IEEE Conference on Computer Vision and Pattern Recognition ( CVPR )

    work page 2023

  6. [6]

    Brooks, T.; Peebles, B.; Holmes, C.; DePue, W.; Guo, Y.; Jing, L.; Schnurr, D.; Taylor, J.; Luhman, T.; Luhman, E.; Ng, C.; Wang, R.; and Ramesh, A. 2024. Video generation models as world simulators

    work page 2024

  7. [7]

    Chang, Z.; Zhang, X.; Wang, S.; Ma, S.; and Gao, W. 2022. STAM: A SpatioTemporal Attention based Memory for Video Prediction. IEEE Transactions on Multimedia, 1--1

    work page 2022

  8. [8]

    Denton, E.; and Fergus, R. 2018. Stochastic video generation with a learned prior. In International conference on machine learning, 1174--1183. PMLR

    work page 2018

Show all 64 references
  1. [9]

    L.; et al

    Denton, E. L.; et al. 2017. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems, 30

    2017

  2. [10]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  3. [11]

    Mitra , N.; and Vedaldi, A

    Ehrhardt, S.; Groth, O.; Monszpart, A.; Engelcke, M.; Posner, I.; J. Mitra , N.; and Vedaldi, A. 2020. RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces. In Advances in Neural Information Processing Systems (NeurIPS)

    2020

  4. [12]

    R.; Jones, O

    Engelcke, M.; Kosiorek, A. R.; Jones, O. P.; and Posner, I. 2019. Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13052

    2019 arXiv

  5. [13]

    Esser, P.; Rombach, R.; and Ommer, B. 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12873--12883

    2021

  6. [14]

    Gao, Z.; Tan, C.; Wu, L.; and Li, S. Z. 2022. SimVP: Simpler Yet Better Video Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3170--3180

    2022

  7. [15]

    J.; Gnanapragasam, D.; Golemo, F.; Herrmann, C.; Kipf, T.; Kundu, A.; Lagun, D.; Laradji, I.; Liu, H.-T

    Greff, K.; Belletti, F.; Beyer, L.; Doersch, C.; Du, Y.; Duckworth, D.; Fleet, D. J.; Gnanapragasam, D.; Golemo, F.; Herrmann, C.; Kipf, T.; Kundu, A.; Lagun, D.; Laradji, I.; Liu, H.-T. D.; Meyer, H.; Miao, Y.; Nowrouzezahrai, D.; Oztireli, C.; Pot, E.; Radwan, N.; Rebain, D....

    2022

  8. [16]

    Gupta, A.; Tian, S.; Zhang, Y.; Wu, J.; Mart \' n-Mart \' n, R.; and Fei-Fei, L. 2022. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.11894

    2022 arXiv

  9. [17]

    He, K.; Gkioxari, G.; Doll \'a r, P.; and Girshick, R. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, 2961--2969

    2017

  10. [18]

    Henderson, P.; and Lampert, C. H. 2020. Unsupervised object-centric video generation and decomposition in 3D . In Advances in Neural Information Processing Systems (NeurIPS) 33

    2020

  11. [19]

    H.; and Bickel, B

    Henderson, P.; Lampert, C. H.; and Bickel, B. 2021. Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure. arXiv:2106.09051

    2021 arXiv

  12. [20]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 6840--6851

    2020

  13. [21]

    Ho, J.; Salimans, T.; Gritsenko, A.; Chan, W.; Norouzi, M.; and Fleet, D. J. 2022. Video Diffusion Models. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 8633--8646. Curran Associates, Inc

    2022

  14. [22]

    H \"o ppe, T.; Mehrjou, A.; Bauer, S.; Nielsen, D.; and Dittadi, A. 2022. Diffusion models for video prediction and infilling. arXiv preprint arXiv:2206.07696

    2022 arXiv

  15. [23]

    Horé, A.; and Ziou, D. 2010. Image Quality Metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition, 2366--2369

    2010

  16. [24]

    F.; and Niebles, J

    Hsieh, J.-T.; Liu, B.; Huang, D.-A.; Fei-Fei, L. F.; and Niebles, J. C. 2018. Learning to Decompose and Disentangle Representations for Video Prediction. In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., Advances in Neural Inform...

    2018

  17. [25]

    Jiang, J.; Janghorbani, S.; De Melo, G.; and Ahn, S. 2019. SCALOR: Generative World Models with Scalable Object Representations. In International Conference on Learning Representations

    2019

  18. [26]

    Jiang, L.; Dai, B.; Wu, W.; and Loy, C. C. 2021. Focal Frequency Loss for Image Reconstruction and Synthesis. In ICCV

    2021

  19. [27]

    F.; Mahendran, A.; Stone, A.; Sabour, S.; Heigold, G.; Jonschkowski, R.; Dosovitskiy, A.; and Greff, K

    Kipf, T.; Elsayed, G. F.; Mahendran, A.; Stone, A.; Sabour, S.; Heigold, G.; Jonschkowski, R.; Dosovitskiy, A.; and Greff, K. 2021. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594

    2021 arXiv

  20. [28]

    Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25

    2012

  21. [29]

    Lee, D.; Lee, J.; and Choi, J. 2023. CAST: Cross-Attention in Space and Time for Video Action Recognition. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, volume 36, 79399--79425. Curran Ass...

    2023

  22. [30]

    Y.; Huang, T

    Lee, W.; Jung, W.; Zhang, H.; Chen, T.; Koh, J. Y.; Huang, T. E.; Yoon, H.; Lee, H.; and Hong, S. 2021. Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction. CoRR, abs/2104.06697

    2021 arXiv

  23. [31]

    A.; Hu, W.; Sun, Z.; and Fisher, R

    Li, N.; Raza, M. A.; Hu, W.; Sun, Z.; and Fisher, R. 2021. Object-Centric Representation Learning with Generative Spatial-Temporal Factorization. In Ranzato, M.; Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Processing System...

    2021

  24. [32]

    Lin, X.; Li, Y.; Hsiao, J.; Ho, C.; and Kong, Y. 2023. Catch Missing Details: Image Reconstruction with Frequency Augmented Variational Autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2023

  25. [33]

    Liu, Y.; Zhang, K.; Li, Y.; Yan, Z.; Gao, C.; Chen, R.; Yuan, Z.; Huang, Y.; Sun, H.; Gao, J.; et al. 2024. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177

    2024 arXiv

  26. [34]

    Locatello, F.; Weissenborn, D.; Unterthiner, T.; Mahendran, A.; Heigold, G.; Uszkoreit, J.; Dosovitskiy, A.; and Kipf, T. 2020. Object-centric learning with slot attention. Advances in neural information processing systems, 33: 11525--11538

    2020

  27. [35]

    Lüddecke, T.; and Ecker, A. 2022. Image Segmentation Using Text and Image Prompts. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7076--7086

    2022

  28. [36]

    Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I.; et al. 2018. Improving language understanding by generative pre-training

    2018

  29. [37]

    Ravuri, S.; Lenc, K.; Willson, M.; Kangin, D.; Lam, R.; Mirowski, P.; Fitzsimons, M.; Athanassiadou, M.; Kashem, S.; Madge, S.; et al. 2021. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597(7878): 672--677

    2021

  30. [38]

    Reis, D.; Kupec, J.; Hong, J.; and Daoudi, A. 2023. Real-time flying object detection with YOLOv8. arXiv preprint arXiv:2305.09972

    2023 arXiv

  31. [39]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10684--10695

    2022

  32. [40]

    S.; Duckworth, D.; Mahendran, A.; Van Steenkiste, S.; Pavetic, F.; Lucic, M.; Guibas, L

    Sajjadi, M. S.; Duckworth, D.; Mahendran, A.; Van Steenkiste, S.; Pavetic, F.; Lucic, M.; Guibas, L. J.; Greff, K.; and Kipf, T. 2022. Object scene representation transformer. Advances in neural information processing systems, 35: 9512--9524

    2022

  33. [41]

    Schmeckpeper, K.; Georgakis, G.; and Daniilidis, K. 2021. Object-centric video prediction without annotation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 13604--13610. IEEE

    2021

  34. [42]

    Schuldt, C.; Laptev, I.; and Caputo, B. 2004. Recognizing human actions: a local SVM approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, 32--36. IEEE

    2004

  35. [43]

    M.; Pugeault, N.; and Henderson, P

    Shi, T.; Ge, X.; Jose, J. M.; Pugeault, N.; and Henderson, P. 2024. Detail-Enhanced Intra-and Inter-modal Interaction for Audio-Visual Emotion Recognition. arXiv preprint arXiv:2405.16701

    2024 arXiv

  36. [44]

    Shi, X.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-K.; and Woo, W.-c. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28

    2015

  37. [45]

    Singh, G.; Wu, Y.-F.; and Ahn, S. 2022. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems, 35: 18181--18196

    2022

  38. [46]

    Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; and Ganguli, S. 2015. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Bach, F.; and Blei, D., eds., Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine ...

    2015

  39. [47]

    Sun, M.; Wang, W.; Zhu, X.; and Liu, J. 2023. MOSO: Decomposing MOtion, Scene and Object for Video Prediction. arXiv:2303.03684

    2023 arXiv

  40. [48]

    van den Oord, A.; Vinyals, O.; and Kavukcuoglu, K. 2017. Neural Discrete Representation Learning. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 30. Curran As...

    2017

  41. [49]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  42. [50]

    Villar-Corrales, A.; Wahdan, I.; and Behnke, S. 2023. Object-Centric Video Prediction Via Decoupling of Object Dynamics and Interactions. In 2023 IEEE International Conference on Image Processing (ICIP), 570--574

    2023

  43. [51]

    Wang, X.; Zhu, Z.; Huang, G.; Wang, B.; Chen, X.; and Lu, J. 2024. Worlddreamer: Towards general world models for video generation via predicting masked tokens. arXiv preprint arXiv:2401.09985

    2024 arXiv

  44. [52]

    Wang, Y.; Gao, Z.; Long, M.; Wang, J.; and Philip, S. Y. 2018. Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International Conference on Machine Learning, 5123--5132. PMLR

    2018

  45. [53]

    Wang, Y.; Wu, H.; Zhang, J.; Gao, Z.; Wang, J.; Yu, P.; and Long, M. 2022. PredRNN: A recurrent neural network for spatiotemporal predictive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2022

  46. [54]

    Wang, Z.; Bovik, A.; Sheikh, H.; and Simoncelli, E. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600--612

    2004

  47. [55]

    Wu, J.; Yin, S.; Feng, N.; He, X.; Li, D.; Hao, J.; and Long, M. 2024. iVideoGPT: Interactive VideoGPTs are Scalable World Models. arXiv preprint arXiv:2405.15223

    2024 arXiv

  48. [56]

    Wu, Z.; Dvornik, N.; Greff, K.; Kipf, T.; and Garg, A. 2022. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861

    2022 arXiv

  49. [57]

    Xiong, R.; Yang, Y.; He, D.; Zheng, K.; Zheng, S.; Xing, C.; Zhang, H.; Lan, Y.; Wang, L.; and Liu, T. 2020. On layer normalization in the transformer architecture. In International Conference on Machine Learning, 10524--10533. PMLR

    2020

  50. [58]

    Yan, W.; Zhang, Y.; Abbeel, P.; and Srinivas, A. 2021. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157

    2021 arXiv

  51. [59]

    Yang, J.; Gao, S.; Qiu, Y.; Chen, L.; Li, T.; Dai, B.; Chitta, K.; Wu, P.; Zeng, J.; Luo, P.; Zhang, J.; Geiger, A.; Qiao, Y.; and Li, H. 2024. Generalized Predictive Model for Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  52. [60]

    Yi, K.; Gan, C.; Li, Y.; Kohli, P.; Wu, J.; Torralba, A.; and Tenenbaum, J. B. 2019. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442

    2019 arXiv

  53. [61]

    Yu, S.; Sohn, K.; Kim, S.; and Shin, J. 2023. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 18456--18466

    2023

  54. [62]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586--595

    2018

  55. [63]

    Zhou, Y.; Zhang, H.; Lee, H.; Sun, S.; Li, P.; Zhu, Y.; Yoo, B.; Qi, X.; and Han, J.-J. 2022. Slot-vps: Object-centric representation learning for video panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3093--3103

    2022

  56. [64]

    Zhu, H.; Ke, W.; Li, D.; Liu, J.; Tian, L.; and Shan, Y. 2022. Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4692--4702

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.