Video Representation Learning with Joint-Embedding Predictive Architectures

Katrina Drozdov; Ravid Shwartz-Ziv; Yann LeCun

REVIEW 4 major objections 8 minor 2 cited by

Video Representation Learning with Joint-Embedding Predictive Architectures

T0 review · 4 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A video JEPA that predicts hidden representations instead of pixels, regularized by variance and covariance, yields frozen features that better capture object speeds and future actions than a generative pixel-space baseline.

desk verdict VJ-VCR is a modest but real contribution; the deterministic speed-probing results hold up, but the CATER latent-action experiment is circular and needs to be reframed. read the letter →

arxiv 2412.10925 v1 pith:IJFTKYG2 submitted 2024-12-14 cs.CV cs.AI

Katrina Drozdov , Ravid Shwartz-Ziv , Yann LeCun This is my paper · ORCID

classification cs.CVcs.AI

keywords videorepresentationlearningself-supervisedjoint-embeddingpredictivearchitecturevariance-covarianceregularizationcollapselatentvariablesdynamicsunderstandingactionrecognition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces VJ-VCR, a self-supervised video model that predicts the hidden representation of future frames rather than their pixels. The central claim is that this abstract-space prediction, combined with variance-covariance regularization to prevent collapse, produces frozen representations that capture high-level dynamics such as object speed and action content better than a generative pixel-space baseline. On MovingMNIST speed probing the hidden-space model reaches MSE 0.04 versus 0.10–0.15 for the generative baseline; on CLEVRER the gap is 0.19 versus 0.22–0.23. The paper also argues that latent variables inferred by energy minimization encode information about stochastic future events, as shown by CATER action recognition (mAP 67.4 versus 53.8). A reader would care because it suggests a path to video representations that are both cheaper to train and more useful for downstream dynamic reasoning.

What carries the argument

The central object is the variance-covariance regularizer (VCR): a hinge loss that keeps the variance of each component of the hidden representation above a threshold while penalizing off-diagonal entries of the covariance matrix, applied at each time step of the video embedding. In VJ-VCR this regularizer is applied to the concatenated hidden states $[h_x, h_y]$, and the training objective is an energy function $E(x,y,z) = \|\mathrm{Pred}(h_x, z) - h_y\|^2 + \alpha l_{\mathrm{var}} + \beta l_{\mathrm{cov}} + \gamma \text{reconstruction}$. The predictor, not a decoder, is the module that must forecast the future, and the latent variable $z$ is an optional extra input that the predictor can use to disambiguate stochastic futures. During inference $z^*$ is found by gradient descent (or FISTA for sparse $z$), and the paper evaluates what $z^*$ reveals about the future.

What would settle it

Train the same VJ-VCR on CATER with $z$ sampled from a fixed random distribution during pre-training instead of the ground-truth action vector, then infer $z^*$ by gradient descent and measure mAP on action recognition. If mAP drops to near the 39.6% random-latent baseline, the current result does not demonstrate that the representation encodes stochastic dynamics; if it stays near 67%, the latent-variable claim is supported.

Watch

Extended reading notes

Core claim

The paper's core discovery is that a joint-embedding predictive architecture, which predicts in representation space and uses variance-covariance regularization, prevents collapse and learns representations that are more informative about dynamics than those from a generative model trained to reconstruct pixels. Over the same encoder and predictor, the VJ-VCR objective is a weighted sum of an energy term (MSE between predicted and true target embeddings) and the VCR regularizer, with no pixel reconstruction required. The authors show that adding a decoder and reconstruction loss to VJ-VCR preserves the speed-probing advantage while improving reconstruction quality (PSNR 21.2 vs 19.5), whereas pure pixel-space models with or without VCR have better reconstruction but worse speed probing. They further find that latent variables incorporated into the predictor can carry stochastic information: a discrete or sparse latent predicts the random trajectory switch in MovingMNIST up to 99.5% accuracy, and a latent inferred by gradient descent on CATER supports multi-label action recognition at 67.4% mAP, 13.6 points above the generative baseline and well above the 39.6% random-latent floor. The paper frames these results as evidence that prediction in abstract representation space focuses learning on high-level, dynamics-relevant information rather than low-level pixel detail.

Load-bearing premise

The claim that latent variables capture stochastic future information rests on the assumption that the action-recognition evaluation measures the learned representation's understanding of dynamics, not simply the inference procedure recovering the ground-truth action vectors that were provided to the predictor during pre-training.

Editorial extensions

If this is right

Frozen VJ-VCR representations can be linearly probed for object speed with substantially lower error than generative baselines, implying the hidden space encodes dynamics without fine-tuning.
Adding a reconstruction decoder to a JEPA does not degrade dynamics understanding; it can preserve speed-probing performance while improving pixel-level reconstruction.
Variance-covariance regularization prevents collapse without negative samples, momentum encoders, or stop-gradient architectural asymmetry.
Latent variables can decompose video prediction into deterministic and stochastic parts, with the inferred latent $z^*$ carrying information about future actions or trajectory switches.
Because predictions happen in hidden space, the model avoids generating pixel-level details, which may reduce compute for high-dimensional video.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If VJ-VCR scales beyond synthetic datasets, hidden-space prediction could become a cheaper alternative to video autoencoders for downstream control and planning, since dynamics-relevant features are learned without reconstruction.
The action-recognition result should be tested without ground-truth action labels during pretraining; if the advantage persists with a randomly sampled $z$, the learned representations themselves encode stochastic dynamics rather than the inference procedure recovering training labels.
The sparse-latent experiments suggest that sparsity level can control the trade-off between stochastic and static information; a similar knob might be used for controllable video prediction or to disentangle content and motion.
The singular-value analysis implies that VCR keeps the representation high-dimensional; one testable extension is whether multi-layer VCR would further improve the rank and downstream transfer.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The paper does one new thing and does it cleanly: it combines VICReg's variance-covariance regularization with a video JEPA, showing that you can avoid collapse without momentum encoders, stop-gradients, or negative samples. The deterministic speed-probing experiments on MovingMNIST and CLEVRER are simple but well-posed, and the gap over the generative baseline (0.04 vs 0.10/0.15 MSE; 0.19 vs 0.22/0.23) supports the central claim that prediction in representation space retains more dynamics information than pixel-space prediction. That part deserves credit.

The soft spots are real, though. The CATER action-recognition experiment is the one I'd push back on. During pretraining the latent z is set to the ground-truth action vector and fed to the predictor; at inference z* is obtained by minimizing the energy relative to the target hidden state. A linear probe on z* then measures how well the inference procedure can retrieve the training-time labels, not whether the learned representation understands dynamics. The 67.4 vs 53.8 mAP gap may just reflect that VJ-VCR's predictor is easier to invert. It does not establish the paper's broad claim about capturing stochastic future information. The authors should either redesign the experiment so the latent isn't supervised during pretraining or explicitly reframe it as a study of latent-variable inference, not representation quality.

Also, Table 1 has no error bars or seed counts, there's no code release, and the only comparator is a generative baseline with the same backbone. These are fixable but necessary for the numbers to be convincing. RankMe is reported but not well integrated into the argument.

The paper is honest about its synthetic-dataset limitation and the writing is clear. The central deterministic result is solid enough to take seriously. This is a conditional accept kind of paper: with repeated seeds, code, and a repaired CATER evaluation, it would be a useful contribution.

Referee Report

4 major / 8 minor

Summary. The paper proposes VJ-VCR, a joint-embedding predictive architecture for self-supervised video representation learning, in which an encoder maps past and future frames to hidden representations and a predictor predicts the future hidden state from the past hidden state. Collapse is prevented by VICReg-style variance and covariance regularization applied to the hidden representations. The paper also studies optional latent variables z intended to capture stochastic information about the future, with discrete, sparse, and continuous formulations, and reports experiments on MovingMNIST, CLEVRER, and CATER. The main empirical claims are that VJ-VCR representations outperform those of a generative pixel-space baseline on linear speed probing, that inferred latent variables capture information about random trajectory switches and object actions, and that VJ-VCR avoids dimensional collapse better than the generative baseline.

Significance. If the empirical claims hold, the paper makes a useful contribution by showing that a simple regularization-based anti-collapse mechanism suffices for video JEPA training and that representation-space prediction yields features that are more informative about dynamics than pixel-space prediction. The use of publicly available code (OpenSTL) and synthetic benchmarks aids reproducibility. The speed-probing results in Table 1 are plausible and the sparse-latent MovingMNIST experiment is an interesting demonstration that an unsupervised latent can encode genuinely stochastic information. However, the significance is moderated by the reliance on small synthetic datasets, the absence of comparisons to established video JEPA methods such as V-JEPA, and by the confounded CATER experiment, which limits the strength of the claim about latent variables and high-level dynamics.

major comments (4)

[Section 4.3.2 / Section 5.2, Figure 2, Eqs. (5)-(6)] The CATER action-recognition experiment does not probe the learned video representation. During pretraining, the latent z is set to the ground-truth action vector, so the predictor is explicitly trained to map (h_x, a_y) to h_y. At inference, z* is obtained by minimizing the energy in Eq. (6), and the linear classifier then operates on z*. The reported mAP gap (67.4 vs 53.8) therefore measures how easily the energy landscape can be inverted to recover the supervised latent label, not the quality of h_x or h_y as dynamical representations. This undercuts the paper's claim that this experiment supports the hypothesis that hidden representations capture high-level dynamics. Please either re-frame the CATER result as a property of the predictor/energy function, or add a direct probe of the learned representation (e.g., linear action recognition from h_y) and a latent-free baseline.
[Table 1] The central speed-probing comparisons are reported without error bars, confidence intervals, or seed counts. The differences (MovingMNIST MSE 0.04 vs 0.10/0.15; CLEVRER 0.19 vs 0.22/0.23) are the primary evidence for the claim that VJ-VCR representations outperform generative-baseline representations, yet without repeated-seed statistics it is impossible to assess whether these gaps are significant. Please report mean ± standard deviation over at least three random seeds for every entry in Table 1, and similarly for Table 2.
[Section 4.3.2, Table 2] For the discrete latent experiment on stochastic MovingMNIST, z is a one-hot encoding of the ground-truth trajectory switch ψ and the active component selects a predictor layer; thus the 79.7% accuracy is again an inversion of the energy to recover a label that was provided during pretraining, and the result is partly circular. The sparse latent experiment is potentially a cleaner test, but the manuscript does not state whether the ground-truth switch is used to supervise the sparse z during training, or whether it is learned purely from the prediction objective. Please clarify the supervision protocol for the sparse latent, and either remove the discrete result or explicitly discuss the invertibility confound.
[Appendix B] All hyperparameters are chosen by grid search on the validation loss, and all reported metrics are also measured on the validation set. This selection procedure risks overfitting the validation set and can bias the comparison between methods. Please either evaluate on a separate held-out split or describe a nested model-selection procedure that does not use the evaluation split for hyperparameter tuning.

minor comments (8)

[Section 3.1] There is a typo: 'representaiton' should be 'representation'.
[Section 4.4 and Appendix B.1] The dataset name 'CLEVRER' is misspelled as 'CLEVERER' in several places, including Section 4.4 and Appendix B.1.
[Appendix B] The appendix heading 'Evalutation Details' contains a typo; it should read 'Evaluation Details'.
[Figure 4] The caption for panel (a) says singular values are measured 'at the beginning and the end of training', but panel (b) is described as showing 'beginning, middle, and end'. Please make the description consistent.
[Section 6] The sentence 'The singular values of the VJ-VCR model are more uniformly distributed than those of coming from the generative-based model' has a grammatical error; 'those of coming from' should be 'those coming from'.
[Section 7] The conclusion states that variance-covariance regularization is applied 'solely to the top layer of the encoder', but Section 3.2 and Eq. (5) apply VCR to the combined set [h_x, h_y]. Please clarify which layer or layers are regularized in the actual implementation.
[Figure 2] The 'Random' baseline is described only as 'randomly generated latent variables z*'. Please specify the distribution and dimensions used to generate these random latents, since this is the reference point for interpreting the mAP values.
[Section 5.2, Figure 2] The text reports that both VJ-VCR and generative results have standard deviation smaller than 2×10−2 over three seeds, but no error bars are shown in Figure 2. Please include error bars or report the per-seed values.

Circularity Check

1 steps flagged · score 6.0 of 10

CATER action-recognition result is partly circular: the latent variable is initialized to ground-truth actions during pretraining, so inferring z* and probing it for those actions largely retrieves the supervision label rather than testing unsupervised dynamics understanding.

fitted input called prediction [Section 4.3.2, CATER paragraph (Figure 2)]
"In particular, during pre-training, the latent z is a binary vector that provides the ground truth actions across all time steps to the predictor. During inference, for each sample, we use several iterations of gradient descent to compute z∗, an approach similar to the algorithm described in Henaff et al. (2017). We then evaluate whether the inferred latent variable z∗ captures the aggregated set of ground-truth actions ay present in the target frames through linear probing for the task of multi-label action recognition."

The CATER latent variable is not learned from the video; it is set to the ground-truth action vector ay during pretraining and fed to the predictor, which is trained to map (hx, ay) to the target hidden state hy. At inference, z* is obtained by minimizing ||Pred(hx,z)-hy||^2, so it can recover ay by inverting the trained predictor. A linear probe on z* then 'predicts' actions that were already supplied as supervision. The 67.4 vs 53.8 mAP gap therefore measures how invertible the energy landscape is with respect to the supervised label, not whether the learned representations encode action information.

full rationale

Most of the paper is self-contained: VJ-VCR's variance-covariance regularization is adapted from VICReg with stated equations, and the speed-probing evaluations on deterministic MovingMNIST and CLEVRER compare frozen representations from JEPA-style versus generative losses without injecting labels; those results support the representation claim. The SVD/RankMe analyses are also independent. The circular element is confined to the CATER action-recognition experiment in Section 4.3.2, where the latent variable used during pretraining is the ground-truth action vector. Inferring z* by energy minimization and then linearly probing it for those same actions reduces, at least partly, to retrieving the supervision signal that was fed into the predictor. This makes the CATER mAP comparison a test of energy-landscape invertibility rather than of unsupervised dynamics understanding. Because the abstract and contributions present latent-variable uncertainty capture as a key contribution, this partial circularity is substantive. However, the central representation-learning claim is independently supported by speed probing, so the overall score is 6 rather than higher.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claims rest on standard machinery for self-supervised representation learning plus several domain choices. The main fitted quantities are the VCR coefficients alpha and beta, chosen by grid search on validation loss, and the sparsity regularization controlling the latent variable. The biggest conceptual assumption is that linear probing on frozen features is an adequate readout of high-level dynamics information. The CATER experiment additionally assumes that supplying ground-truth actions as the latent during pretraining does not invalidate the later inference-based action recognition. No invented physical entities are introduced; the latent variable z is a modeling construct with no independent falsifiable handle in this paper.

free parameters (3)

variance regularization coefficient alpha = 0.5 for MovingMNIST, 1 for CLEVRER and CATER
Set by grid search on validation loss per Appendix B.1; directly controls the VCR objective in Eq. (4).
covariance regularization coefficient beta = 0.1 for MovingMNIST and CLEVRER, 0.4 for CATER
Set by grid search on validation loss per Appendix B.1; directly controls the VCR objective in Eq. (4).
sparsity regularization strength for the sparse latent z = Two levels described as high (80%) and low (20%) average sparsity; exact penalty values not reported
Chosen by experiment and controls the trade-off between encoding stochastic and static information in Table 2.

assumptions (3)

domain assumption Linear probing of frozen representations is a valid measure of the information available for downstream dynamics tasks.
Speed regression and action classification from frozen features assume that high-level semantics are linearly decodable; the paper does not test other readouts.
domain assumption The generative baseline is a controlled comparator, differing only in prediction space and the presence of a decoder.
Section 4.1 states the baseline shares building blocks, but separate hyperparameter tuning and the extra decoder make this control assumption untested.
ad hoc to paper Providing ground-truth actions as latent z during CATER pretraining is compatible with claiming that z captures uncertainty.
Section 4.3.2 supplies the answer to the later action-recognition probe during training, which conflates supervised label recovery with learned representation of stochastic futures.

invented entities (1)

Latent variable z (discrete or sparse) for stochastic future events
purpose: Encodes information about target frames not inferable from input frames, especially trajectory switches in MovingMNIST and actions in CATER.
The paper does not provide an external falsifiable handle for z beyond the same datasets used to train it. For CATER, z is initialized from ground-truth action labels during pretraining, so its independent value is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Video Representation Learning with Joint-Embedding Predictive Architectures." pith.science (2026). https://pith.science/paper/IJFTKYG2

@misc{pith2026241210925,
  author       = {Pith},
  title        = {Pith review of: Video Representation Learning with Joint-Embedding Predictive Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJFTKYG2}},
  note         = {Machine review of arXiv:2412.10925}
}

read the original abstract

Video representation learning is an increasingly important topic in machine learning research. We present Video JEPA with Variance-Covariance Regularization (VJ-VCR): a joint-embedding predictive architecture for self-supervised video representation learning that employs variance and covariance regularization to avoid representation collapse. We show that hidden representations from our VJ-VCR contain abstract, high-level information about the input data. Specifically, they outperform representations obtained from a generative baseline on downstream tasks that require understanding of the underlying dynamics of moving objects in the videos. Additionally, we explore different ways to incorporate latent variables into the VJ-VCR framework that capture information about uncertainty in the future in non-deterministic settings.

Figures

Figures reproduced from arXiv: 2412.10925 by the authors.

**Figure 1.** We evaluate our models by predicting the aggregate actions [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

**Figure 2.** Multi-label action recognition performed on the CATER dataset. The aggregated set of actions [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Reconstructions from our VJ-VCR model trained with and without a latent variable on the [PITH_FULL_IMAGE:figures/full_fig_p012_3.png]

**Figure 4.** Figure 4: Analysis of the informational content of the learned hidden representations of a VJ-VCR and a [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: Reconstructions from a generative model trained only with loss in pixel space (left) and a VJ-VCR [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions
cs.LG 2025-05 reject novelty 6.0 of 10

CHARM is a 7M-parameter self-supervised embedding model for multivariate time series that uses channel descriptions to beat specialized baselines on forecasting, classification, and anomaly detection.
A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
cs.RO 2025-07 conditional novelty 4.0 of 10

Embodied intelligence learning is reviewed through the complementary lenses of physical simulators and world models, with a proposed IR-L0 to IR-L4 robot capability taxonomy.

Reference graph

Works this paper leans on

16 extracted references · 8 canonical work pages · cited by 2 Pith papers

[6]

Adam: A method for stochastic optimization

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

arXiv
[10]

Evolving Losses for Unlabeled Video Representation Learning

AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unlabeled video representation learning. arXiv preprint arXiv:1906.03248 ,

work page Pith review arXiv 1906
[11]

An information- theoretic perspective on variance-invariance-covariance regularization.arXiv preprint arXiv:2303.00633 ,

Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim GJ Rudner, and Yann LeCun. An information- theoretic perspective on variance-invariance-covariance regularization.arXiv preprint arXiv:2303.00633 ,

arXiv
[12]

Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442 ,

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442 ,

arXiv 1910
[13]

Variance-covariance regularization improves representation learning.arXiv preprint arXiv:2306.13292 ,

Jiachen Zhu, Katrina Evtimova, Yubei Chen, Ravid Shwartz-Ziv, and Yann LeCun. Variance-covariance regularization improves representation learning.arXiv preprint arXiv:2306.13292 ,

arXiv
[14]

16 A Datasets MovingMNIST For experiments with MovingMNIST, we split the original MNIST dataset into 55,000 training and 5,000 validation samples. In the deterministic version of the MovingMNIST, given a sample from the MNIST dataset, we generate a 20-frame video by randomly sampling the digit’s initial location on a 64× 64 black canvas and the digit’s ve...

work page 2019
[15]

Each video contains 300 frames at 24 frames per second at 320x240 resolution

is a synthetic dataset of moving objects that can move independently and also interact with each other. Each video contains 300 frames at 24 frames per second at 320x240 resolution. There are 14 possible actions objects can perform and multiple actions can be present in a single video. In our experiments, we reshape the video to128× 128 resolution. We use...

work page 2019
[16]

The model on the left has PSNR of 22.8 and the one on the right has PSNR of 21.2

reconstructed frames. The model on the left has PSNR of 22.8 and the one on the right has PSNR of 21.2. Both models can predict the trajectories of the digits. Hidden representations from the VJ-VCR model can be used to predict the actual speed of the digits more accurately. per second following Girdhar & Ramanan (2019). In VJ-VCR experiments with CATER, ...

work page 2019

Show all 16 references

[1993]

neurips.cc/paper_files/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf

URLhttps://proceedings. neurips.cc/paper_files/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In...

1993
[2014]

A path towards autonomous machine intelligence version 0.9

Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.Open Review, 62 (1),

2022
[2015]

A comprehensive review on autonomous navigation.arXiv preprint arXiv:2212.12808,

Saeid Nahavandi, Roohallah Alizadehsani, Darius Nahavandi, Shady Mohamed, Navid Mohajer, Mohammad Rokonuzzaman, and Ibrahim Hossain. A comprehensive review on autonomous navigation.arXiv preprint arXiv:2212.12808,

arXiv
[2017]

Understanding dimensional collapse in contrastive self-supervised learning

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348 ,

arXiv
[2019]

Prediction under uncertainty with error-encoding networks

Mikael Henaff, Junbo Zhao, and Yann LeCun. Prediction under uncertainty with error-encoding networks. arXiv preprint arXiv:1711.04994 ,

arXiv
[2020]

Dimensionality reduction by learning an invariant mapping

14 Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06) , volume 2, pp. 1735–1742. IEEE,

2006
[2021]

Deep multi-scale video prediction beyond mean square error

Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440 ,

arXiv
[2023]

Cater: A diagnostic dataset for compositional actions and temporal reasoning

Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. arXiv preprint arXiv:1910.04744 ,

1910 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [6]

Adam: A method for stochastic optimization

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

arXiv

[2] [10]

Evolving Losses for Unlabeled Video Representation Learning

AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unlabeled video representation learning. arXiv preprint arXiv:1906.03248 ,

work page Pith review arXiv 1906

[3] [11]

An information- theoretic perspective on variance-invariance-covariance regularization.arXiv preprint arXiv:2303.00633 ,

Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim GJ Rudner, and Yann LeCun. An information- theoretic perspective on variance-invariance-covariance regularization.arXiv preprint arXiv:2303.00633 ,

arXiv

[4] [12]

Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442 ,

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442 ,

arXiv 1910

[5] [13]

Variance-covariance regularization improves representation learning.arXiv preprint arXiv:2306.13292 ,

Jiachen Zhu, Katrina Evtimova, Yubei Chen, Ravid Shwartz-Ziv, and Yann LeCun. Variance-covariance regularization improves representation learning.arXiv preprint arXiv:2306.13292 ,

arXiv

[6] [14]

16 A Datasets MovingMNIST For experiments with MovingMNIST, we split the original MNIST dataset into 55,000 training and 5,000 validation samples. In the deterministic version of the MovingMNIST, given a sample from the MNIST dataset, we generate a 20-frame video by randomly sampling the digit’s initial location on a 64× 64 black canvas and the digit’s ve...

work page 2019

[7] [15]

Each video contains 300 frames at 24 frames per second at 320x240 resolution

is a synthetic dataset of moving objects that can move independently and also interact with each other. Each video contains 300 frames at 24 frames per second at 320x240 resolution. There are 14 possible actions objects can perform and multiple actions can be present in a single video. In our experiments, we reshape the video to128× 128 resolution. We use...

work page 2019

[8] [16]

The model on the left has PSNR of 22.8 and the one on the right has PSNR of 21.2

reconstructed frames. The model on the left has PSNR of 22.8 and the one on the right has PSNR of 21.2. Both models can predict the trajectories of the digits. Hidden representations from the VJ-VCR model can be used to predict the actual speed of the digits more accurately. per second following Girdhar & Ramanan (2019). In VJ-VCR experiments with CATER, ...

work page 2019