Pith. sign in

REVIEW 4 major objections 6 minor 60 references

EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One transformer learns to see, foresee, and act together.

desk verdict A solid joint training framework with convincing ablations, but the future-state benchmark is self-referential and can't carry the 'predict' claim on its own. read the letter →

arxiv 2502.05857 v3 pith:KOFB53BT submitted 2025-02-09 cs.CV cs.AIcs.LG

Lu Chen , Yizhou Wang , Shixiang Tang , Qianhong Ma , Tong He , Wanli Ouyang , Xiaowei Zhou , Hujun Bao
show 1 more author
Sida Peng
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords egocentricvisionagentmodelworld3Dhumanmotionpredictionjointembeddingpredictivearchitectureself-supervisedlearningvisualrepresentationvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EgoAgent tries to show that one transformer, trained from scratch on egocentric video frames paired with synchronized 3D body poses, can acquire three abilities at once: representing what the camera sees, predicting the next visual world state, and forecasting the person's 3D motion. The proposal is that these abilities should not be trained as separate specialist models, because perception and action are causally intertwined: observations trigger actions, and actions change what is observed next. To capture that loop, the model reads the interaction as an interleaved token sequence of states and actions under causal attention, with a joint embedding-action-prediction objective that aligns predicted future states with an EMA observer branch in a shared semantic space. On benchmarks for egocentric future-state retrieval, ImageNet k-NN classification, 3D human motion prediction, and a simulated manipulation task, the jointly trained model reports gains over single-task and specialist methods, and ablations show that dropping any one loss hurts the other two abilities. The load-bearing consequence is that world-modeling, perception, and action prediction reinforce one another inside one network, rather than being separable capabilities.

What carries the argument

The machinery is the joint embedding-action-prediction (JEAP) architecture with temporally asymmetric predictor and observer branches. Learnable query tokens $q_a$ and $q_s$ are placed after image and action tokens so that causal attention makes $q_a$ predict the next action from all past observations and $q_s$ predict the next world state from observations plus the just-predicted action; separate MLPs map these embeddings to 3D poses and semantic state features. The observer branch, an exponential-moving-average copy that sees only images, produces the target future-state embedding used by the state-prediction loss, so the model is trained in a continuous semantic feature space rather than a pixel-reconstruction space. The full objective combines a DINO representation loss, a DINO future-state prediction loss, and an L1 action loss, and the ablation study attributes the reported gains to their joint optimization.

What would settle it

Take the trained observer and use its predicted next-frame embeddings to retrieve frames from a held-out egocentric dataset or from the same scenes after a temporal gap; if retrieval accuracy falls to chance while within-dataset accuracy stays high, the apparent foresight is dataset-specific. Alternatively, train a linear decoder from predicted state embeddings to the pixel content of the next frame: failure to recover the actual next image would indicate the predicted 'world state' is not grounded in observable visual change.

Watch

Extended reading notes

Core claim

The central claim is that a single agent model can simultaneously learn to represent egocentric observations, predict future world states, and generate 3D actions, and that these three abilities mutually reinforce each other. EgoAgent encodes observations and poses into continuous semantic embeddings, arranges them as interleaved image, action, and query tokens, and uses causal attention to predict the next action from past observations and the next world state from observations plus the current action. A momentum observer branch, updated as an EMA of the predictor, supplies the target future-state embedding with stop-gradient; the predictor is trained to match that target with a DINO loss, to match ground-truth poses with an L1 loss, and to build representations with a self-supervised view-matching loss. Reported results include a +16.28 Top1 and +16.95 mAP gain over prior egocentric representation models on Ego-Exo4D future-state retrieval, higher ImageNet k-NN accuracy than the strongest egocentric pretraining baseline, and lower MPJPE than dedicated motion-prediction models. The paper also reports that removing any of the three losses degrades the other two tasks, which is the direct evidence for the mutual-reinforcement claim.

Load-bearing premise

The future-state prediction target is defined by the observer branch, an EMA copy of the network itself, so the entire claim rests on that internal feature space being a faithful and discriminative encoding of the visual world rather than a self-consistent but idiosyncratic representation.

Editorial extensions

If this is right

  • A single network trained on egocentric video plus synchronized 3D poses can serve as both a visual encoder and a motion forecaster, removing the need for separate specialist models on these tasks.
  • Jointly optimizing representation, prediction, and action losses improves each individual benchmark relative to training any one task alone, so future agent models should couple these objectives rather than compose pretrained modules.
  • Because conditioning on different 3D poses changes the predicted future state, the learned state space encodes action-consequence relationships, not just visual similarity.
  • Scaling the joint model from 300M to 1B parameters improves future-state retrieval, ImageNet classification, and motion prediction simultaneously.
  • Pixel-level latent spaces such as a VQGAN tokenizer underperform continuous semantic embeddings for this joint objective, so high-level semantics should be the training space for egocentric world models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper leaves implicit: evaluate the observer's predicted next-frame features against galleries from held-out scenes, unseen subjects, or other egocentric datasets; strong transfer would confirm genuine world dynamics rather than dataset-specific shortcuts.
  • The common-coding motivation suggests a testable extension: adding richer action modalities such as gaze, hand pose, or language instructions to the same interleaved-token recipe should further sharpen downstream representation accuracy, since the model would have more action-consequence pairings to align.
  • If the mutual-reinforcement result transfers, agent architectures for robotics could be trained from human egocentric video alone to bootstrap visuomotor policies, reducing reliance on teleoperated robot demonstrations.
  • A reader could test whether the predicted state embeddings are genuinely anticipatory: if a linear decoder can recover the pixel content of the next frame from the predicted embedding, that would confirm they carry real future visual information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes EgoAgent, a transformer trained from scratch on egocentric video and synchronized 3D human poses. The input is formulated as an interleaved sequence of image tokens, action tokens, and learnable query tokens, and a predictor branch is paired with a momentum EMA observer branch. The training objective in Eq. (4) combines a DINO-style representation loss, a feature-space future-state prediction loss, and an L1 action loss. The authors evaluate EgoAgent on egocentric future-state retrieval, 3D human motion prediction, ImageNet k-NN classification, and the TriFinger manipulation benchmark, and report that joint training improves all three abilities relative to single-task variants and to prior methods.

Significance. If the central claim is accepted, the paper is a useful contribution: it gives a concrete architecture (Sec. 3.1 and supplementary Tables 3-4), trains from scratch without pretrained LLM weights, and includes leave-one-out ablations specifically designed to probe cross-task synergy. The commitment to release code and trained models is a strength. However, the main evidence for the 'predict' component rests on a retrieval benchmark evaluated inside the model's own EMA feature space, and the ablation results are single-run point estimates. These issues need to be resolved before the mutual-reinforcement claim is fully supported.

major comments (4)
  1. [§4.2, Eq. (2)] The future-state prediction benchmark is self-referential. The target S_{t+1} in Eq. (2) is produced by the momentum observer Theta_obs, and Section 4.2 embeds the gallery frames with the same EgoAgent feature extractor. Top1 and mAP therefore measure how well the predictor output agrees with its own EMA target encoder, not whether the predicted embedding corresponds to an externally defined future world state. This is the only task where EgoAgent shows a large margin over prior methods in Table 1 (+12.86 Top1 and +13.05 mAP over DoRA at 300M), so the concern is load-bearing. Please add an external anchor, for example freezing an independently trained encoder for the gallery, measuring linear probe accuracy on predicted features against an external annotation of future frames, or adding a quantitative pixel-level forecast comparison.
  2. [§4.2, Table 1] The world-state-prediction comparison is against baselines that are not trained to predict future states. The paper itself states that for traditional representation models the 'predicted' state is exactly the current state S_t, so the comparison is between an action-conditioned future-state predictor and non-predictive feature extractors. It is unsurprising that EgoAgent outperforms such baselines by a large margin. To support the claim that the joint architecture is responsible for the gain, please add baselines that are trained to predict next-frame features under the same protocol, e.g., a linear or small transformer predictor on top of DoRA or DINO features with the same action conditioning.
  3. [Table 3] The mutual-reinforcement claim rests on ablation differences that are reported as single-run point estimates. For example, comparing rows (a) and (f), the MPJPE is 14.49 with all losses versus 14.32 with only L_act, and ImageNet-1K Top1 is 22.28 versus 21.56 for only L_rep. Without standard deviations over multiple seeds, these differences may be within run-to-run noise. Please report mean and standard deviation over at least three seeds for the ablations, or at least for the key comparisons in rows (a)-(i).
  4. [§4.3, Table 1] The motion prediction comparison is not controlled for training data and model scale. EgoAgent is trained on WT+Ego-Exo4D with 300M or 1B parameters, whereas siMLPe, HumanMAC, and Diffusion Policy are trained from scratch on Ego-Exo4D only; the text does not state whether those baselines were given the same combined data, the same number of iterations, or comparable capacity. Since the central claim is that joint learning, not additional data or scale, produces the gains, please add a same-data, same-compute comparison or explicitly control for these factors.
minor comments (6)
  1. [§4.2] The sentence 'Note that St is NOT removed when testing EgoAgent' is confusing; please clarify whether the current frame is included in the gallery for all methods and why it is kept as a distractor for EgoAgent.
  2. [Eq. (4)] The summation is over k from 0 to t, but the loss terms are indexed by t rather than k; please fix the notation.
  3. [§3.1] The temporal indexing of the predicted action is ambiguous: q_a is said to predict A'_t at time step t, while the evaluation in Section 4.3 predicts the last 15 frames given 5 input frames; please align the notation with the actual task.
  4. [Figure 2] The rendered mathematical symbols in Figure 2 appear garbled in the PDF; please use vector fonts and check the final rendering.
  5. [Eq. (2)] The DINO loss L_dino is referenced but not defined in the main text; specify how the two-view or multi-crop setup is used for the future-state prediction loss, since L_pred compares features from different timesteps rather than different views.
  6. [Section 1] The 'first agent model' claim is stronger than needed; prior world models such as GAIA-1 and JEPA-based methods also combine perception and prediction, so the specific novelty should be stated as the joint action branch and the from-scratch egocentric training protocol.

Circularity Check

1 steps flagged · score 6.0 of 10

Future-state retrieval is scored against the same EMA-observer embeddings that supervise Lpred, so the large 'world state prediction' margin partly measures self-consistency in a self-defined feature space.

  1. fitted input called prediction [Sec. 3.2, Eq. (2) and Sec. 4.2, World State Prediction]
    "Lpred(t) = Ldino(S′t+1, sg[St+1]) ... The weights of the observer branch, including the MLPs, are updated at each training iteration using an EMA of the predictor's. ... we build a query set that stores the EgoAgent-predicted features of the next frame and a gallery set that contains the features directly extracted from all video frames. At time step t, if the predicted world state S′t+1 in the query set can correctly retrieve the corresponding St+1 in the gallery set, we treat it as a successful prediction."

    The quantity evaluated as 'world state prediction' is the same quantity the model is trained to produce. Eq. (2) defines the training target St+1 as the output of the observer branch, an EMA copy of the predictor, and Lpred aligns S′t+1 with sg[St+1]. The retrieval benchmark then scores S′t+1 by retrieving 'the corresponding St+1 in the gallery set,' where the gallery features are extracted in this same learned feature space. Thus the reported Top1 and mAP measure how well the predictor imitates its own EMA teacher on the next frame, not how well it forecasts an externally specified world state.

full rationale

EgoAgent's action prediction and representation learning are anchored to external supervision (3D poses, ImageNet labels, and TriFinger success rates), so the paper is not globally circular. However, the benchmark that produces the largest reported margin—egocentric future-state prediction—is internally defined. In Sec. 3.2, Eq. (2), the target St+1 is the output of the observer branch, an EMA copy of the predictor; the predictor is trained to match the stop-gradient target. In Sec. 4.2, the gallery contains features directly extracted from all video frames in the same learned feature space, and the query S′t+1 is scored by retrieving the corresponding St+1. This is the same matching task that Lpred optimizes, so the Top1 and mAP values largely report whether the predictor imitates its own teacher in a self-defined semantic space rather than whether it predicts externally defined world states. The 3D motion prediction and visual representation results are independently grounded and keep the central claim partially substantive; what reduces by construction is the future-state prediction component, which is the paper's main evidence for the 'predict' ability. No load-bearing self-citation chain is present.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical entities, forces, or particles are introduced. The query tokens qa and qs are learnable architectural embeddings, not external entities, and no independent evidence is needed beyond training. The paper's assumptions are mostly about the validity of self-supervised targets, the reliability of the 3D pose data, and the use of retrieval accuracy as a proxy for world-state prediction.

free parameters (4)
  • Loss weights lambda_rep, lambda_pred, lambda_act = lambda_rep=2, lambda_pred=1, lambda_act=3
    Hand-chosen weights balancing the three objectives in Eq. (4); the claimed synergy depends on this balance, and no sensitivity analysis is provided.
  • EMA momentum = 0.996
    EMA momentum controls how slowly the observer target updates; it affects the stability of the future-state prediction target in Eq. (2).
  • Sliding window and time steps = 20 frames, T=4, one image per 5 frames
    The interleaved sequence structure that defines the causal dependency modeling; chosen to reduce training cost, and results may depend on it.
  • DINO crop counts = 2 global crops, 6 local crops
    Self-supervised representation loss settings inherited from DINO and DoRA; they affect representation quality and are not independently varied.
assumptions (6)
  • domain assumption Continuous semantic embeddings are a sufficient representation of world states for prediction and retrieval.
    The paper deliberately avoids reconstruction-based latents (Section 3.1) and evaluates prediction only through feature retrieval (Section 4.2).
  • domain assumption Causal attention over interleaved image and action tokens captures the perception-action loop.
    The core design in Section 3.1 assumes that appending qa after image tokens and qs after action tokens causes the queries to integrate the intended history.
  • domain assumption The EMA observer branch provides a stable, non-collapsing target for future state prediction.
    Eq. (2) uses stop-gradient on St+1 from Theta_obs; this is a standard JEPA assumption but is not proven for this multimodal setting.
  • domain assumption Automatically generated 3D poses in Ego-Exo4D are accurate enough to supervise action prediction.
    Supplementary Section 5 states automated annotations are used when manual labels are absent; errors in these poses would directly corrupt the action loss.
  • domain assumption DINO-style self-supervision transfers to egocentric video and ImageNet classification.
    The representation loss uses DINO between two views in Eq. (3); ImageNet k-NN results assume this SSL signal yields linearly separable features.
  • ad hoc to paper Retrieval in the model's own feature space is a valid measure of future state prediction.
    Section 4.2 evaluates Top1 and mAP retrieval of the predicted embedding against observer embeddings of all frames; this is the paper's chosen protocol and is partly self-referential.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds." pith.science (2026). https://pith.science/paper/KOFB53BT

@misc{pith2026250205857,
  author       = {Pith},
  title        = {Pith review of: EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOFB53BT}},
  note         = {Machine review of arXiv:2502.05857}
}
read the original abstract

Learning an agent model that behaves like humans-capable of jointly perceiving the environment, predicting the future, and taking actions from a first-person perspective-is a fundamental challenge in computer vision. Existing methods typically train separate models for these abilities, which fail to capture their intrinsic relationships and prevent them from learning from each other. Inspired by how humans learn through the perception-action loop, we propose EgoAgent, a unified agent model that simultaneously learns to represent, predict, and act within a single transformer. EgoAgent explicitly models the causal and temporal dependencies among these abilities by formulating the task as an interleaved sequence of states and actions. It further introduces a joint embedding-action-prediction architecture with temporally asymmetric predictor and observer branches, enabling synergistic optimization across all three capabilities. Comprehensive evaluations of EgoAgent on representative tasks such as image classification, egocentric future state prediction, and 3D human motion prediction demonstrate the superiority of our method. The code and trained models will be publicly available at https://github.com/zju3dv/EgoAgent.

Figures

Figures reproduced from arXiv: 2502.05857 by the authors.

Figure 1
Figure 1. (a) Humans learn from the world by continuously perceiving the egocentric world, predicting future states, and taking actions to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of EgoAgent. EgoAgent adopts the Joint Embedding-Action-Prediction Architecture to project input [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Retrieval results for egocentric world state prediction. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Retrieval results for egocentric world state prediction [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visualizations of the 3D human motion prediction task. After observing the egocentric frames, EgoAgent can generate accurate [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 1
Figure 1. Figure 1: Comparison of OpenSora V1.1 first-frame-conditioned [PITH_FULL_IMAGE:figures/full_fig_p012_1.png]
Figure 2
Figure 2. Figure 2: Retrieval and generation results for egocentric future state prediction. Correct and wrong retrieval images are marked with green [PITH_FULL_IMAGE:figures/full_fig_p013_2.png]
Figure 3
Figure 3. Figure 3: 3D Human motion prediction results in scenes with minor changes in egocentric observations. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]
Figure 4
Figure 4. Figure 4: Attention map of EgoAgent performing visual represen [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 44 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    A spatio-temporal transformer for 3D human motion prediction

    Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges. A spatio-temporal transformer for 3D human motion prediction. In IEEE International Conference on 3D Vision, pages 565--574, 2021

    work page 2021

  3. [3]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and Fran c ois Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37: 0 58757--58791, 2024

    2024

  4. [4]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619--15629, 2023

    2023

  5. [5]

    Sequential modeling enables scalable learning for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22861--22872, 2024

    work page 2024

  6. [6]

    Neural game engine: Accurate learning of generalizable forward models from pixels

    Chris Bamford and Simon M Lucas. Neural game engine: Accurate learning of generalizable forward models from pixels. In IEEE Conference on Games, pages 81--88, 2020

    work page 2020

  7. [7]

    MC-JEPA : A joint-embedding predictive architecture for self-supervised learning of motion and content features

    Adrien Bardes, Jean Ponce, and Yann LeCun. MC-JEPA : A joint-embedding predictive architecture for self-supervised learning of motion and content features. arXiv preprint arXiv:2307.12698, 2023

    arXiv 2023

  8. [8]

    Revisiting feature prediction for learning visual representations from video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024

    2024

Show all 60 references
  1. [9]

    Grounded cognition

    Lawrence W Barsalou. Grounded cognition. Annual Review of Psychology, 59 0 (1): 0 617--645, 2008

    2008

  2. [10]

    Long-term human motion prediction with scene context

    Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In European Conference on Computer Vision, pages 387--404, 2020

    2020

  3. [11]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv \'e J \'e gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650--9660, 2021

    2021

  4. [12]

    Humanmac: Masked motion completion for human motion prediction

    Ling-Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang, Xiaobo Xia, and Tongliang Liu. Humanmac: Masked motion completion for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9544--9555, 2023

    2023

  5. [13]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597--1607, 2020

    2020

  6. [14]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2023

    2023

  7. [15]

    Learning dynamic relationships for 3D human motion prediction

    Qiongjie Cui, Huaijiang Sun, and Fei Yang. Learning dynamic relationships for 3D human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6519--6527, 2020

    2020

  8. [16]

    ImageNet : A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet : A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248--255. Ieee, 2009

    2009

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Confer...

    2021

  10. [18]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873--12883, 2021

    2021

  11. [19]

    Learning and leveraging world models in visual representation learning

    Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.00504, 2024

    2024 arXiv

  12. [20]

    Ego4D : Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D : Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  13. [21]

    Ego-Exo4D : Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-Exo4D : Understanding skilled human activity from first-and third-person perspectives. In Proceeding...

    2024

  14. [22]

    Back to MLP : A simple baseline for human motion prediction

    Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda-Pineda, and Francesc Moreno-Noguer. Back to MLP : A simple baseline for human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4809--4819, 2023

    2023

  15. [23]

    Recurrent world models facilitate policy evolution

    David Ha and J \"u rgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems, 31, 2018 a

    2018

  16. [24]

    World models

    David Ha and J \"u rgen Schmidhuber. World models. In Thirty-second Annual Conference on Neural Information Processing Systems, 2018 b

    2018

  17. [25]

    Data-efficient large vision models through sequential autoregression

    Zhiwei Hao, Jianyuan Guo, Chengcheng Wang, Yehui Tang, Han Wu, Han Hu, Kai Han, and Chang Xu. Data-efficient large vision models through sequential autoregression. In International Conference on Machine Learning, pages 17572--17596, 2024

    2024

  18. [26]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770--778, 2016

    2016

  19. [27]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729--9738, 2020

    2020

  20. [28]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \'a r, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000--16009, 2022

    2022

  21. [29]

    GAIA-1 : A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-1 : A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023

    2023 arXiv

  22. [30]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 0 18661--18673, 2020

    2020

  23. [31]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62 0 (1): 0 1--62, 2022

    2022

  24. [32]

    Aggregated multi-gans for controlled 3D human motion prediction

    Zhenguang Liu, Kedi Lyu, Shuang Wu, Haipeng Chen, Yanbin Hao, and Shouling Ji. Aggregated multi-gans for controlled 3D human motion prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2225--2232, 2021

    2021

  25. [33]

    VIP : Towards universal visual reward and representation via value-implicit pre-training

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP : Towards universal visual reward and representation via value-implicit pre-training. The Eleventh International Conference on Learning Representations, 2023

    2023

  26. [34]

    Motionaug: Augmentation with physical correction for human motion prediction

    Takahiro Maeda and Norimichi Ukita. Motionaug: Augmentation with physical correction for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6427--6436, 2022

    2022

  27. [35]

    Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems, 36: 0 655--677, 2023

    Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing S...

    2023

  28. [36]

    Weakly-supervised action transition learning for stochastic human motion prediction

    Wei Mao, Miaomiao Liu, and Mathieu Salzmann. Weakly-supervised action transition learning for stochastic human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8151--8160, 2022

    2022

  29. [37]

    On human motion prediction using recurrent neural networks

    Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2891--2900, 2017

    2017

  30. [38]

    Structured world models from human videos

    Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. arXiv preprint arXiv:2308.10901, 2023

    2023 arXiv

  31. [39]

    R3M : A universal visual representation for robot manipulation

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M : A universal visual representation for robot manipulation. Conference on Robot Learning, pages 892--909, 2023

    2023

  32. [40]

    Perception and action planning

    Wolfgang Prinz. Perception and action planning. European Journal of Cognitive Psychology, 9 0 (2): 0 129--154, 1997

    1997

  33. [41]

    Embodied Cognition

    Lawrence Shapiro and Shannon Spaulding. Embodied Cognition . The Stanford Encyclopedia of Philosophy , 2024

    2024

  34. [42]

    InternLM : A multilingual language model with progressively enhanced capabilities, 2023

    InternLM Team. InternLM : A multilingual language model with progressively enhanced capabilities, 2023

    2023

  35. [43]

    A dynamic systems approach to the development of cognition and action

    Esther Thelen and Linda B Smith. A dynamic systems approach to the development of cognition and action. MIT press, 1994

    1994

  36. [44]

    The dynamics of embodiment: A field theory of infant perseverative reaching

    Esther Thelen, Gregor Sch \"o ner, Christian Scheier, and Linda B Smith. The dynamics of embodiment: A field theory of infant perseverative reaching. Behavioral and Brain Sciences, 24 0 (1): 0 1--34, 2001

    2001

  37. [45]

    Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video

    Shashanka Venkataramanan, Mamshad Nayeem Rizve, Jo \ a o Carreira, Yuki M Asano, and Yannis Avrithis. Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video. The Twelfth International Conference on Learning Representations, 2024

    2024

  38. [46]

    PVRED : A position-velocity recurrent encoder-decoder for human motion prediction

    Hongsong Wang, Jian Dong, Bin Cheng, and Jiashi Feng. PVRED : A position-velocity recurrent encoder-decoder for human motion prediction. IEEE Transactions on Image Processing, 30: 0 6096--6106, 2021

    2021

  39. [47]

    Revisiting the transferability of supervised pretraining: an mlp perspective

    Yizhou Wang, Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Donglian Qi, and Wanli Ouyang. Revisiting the transferability of supervised pretraining: an mlp perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9183--9193, 2022

    2022

  40. [48]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2024

  41. [49]

    TriFinger : An open-source robot for learning dexterity

    Manuel W \"u thrich, Felix Widmaier, Felix Grimminger, Joel Akpo, Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, et al. TriFinger : An open-source robot for learning dexterity. Proceedings of Machine Learning Research, 155: 0 1...

    2020

  42. [50]

    Pandora: Towards general world model with natural language actions and video states

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024

    2024 arXiv

  43. [51]

    Forecasting of 3D whole-body human poses with grasping objects

    Haitao Yan, Qiongjie Cui, Jiexin Xie, and Shijie Guo. Forecasting of 3D whole-body human poses with grasping objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1726--1736, 2024

    2024

  44. [52]

    UniSim : Learning interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. UniSim : Learning interactive real-world simulators. The Twelfth International Conference on Learning Representations, 2024

    2024

  45. [53]

    Ego-pose estimation and forecasting as real-time pd control

    Ye Yuan and Kris Kitani. Ego-pose estimation and forecasting as real-time pd control. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10082--10092, 2019

    2019

  46. [54]

    Incorporating physics principles for precise human motion prediction

    Yufei Zhang, Jeffrey O Kephart, and Qiang Ji. Incorporating physics principles for precise human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6164--6174, 2024

    2024

  47. [55]

    GIMO : Gaze-informed human motion prediction in context

    Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C Karen Liu, and Leonidas J Guibas. GIMO : Gaze-informed human motion prediction in context. In European Conference on Computer Vision, pages 676--694, 2022

    2022

  48. [56]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  49. [57]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  50. [58]

    Denoising diffusion autoencoders are unified self-supervised learners

    Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15802--15812, 2023

    2023

  51. [59]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  52. [60]

    Open-Sora : Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora : Democratizing efficient video production for all, 2024

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.