Pith. sign in

REVIEW 4 major objections 4 minor 138 references

EgoM2P: Egocentric Multimodal Multitask Pretraining

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EgoM2P is a single masked-pretrained Transformer that predicts gaze, head-camera trajectory, monocular depth, and egocentric video from RGB, matching or beating specialist models while running an order of magnitude faster.

desk verdict Solid engineering, honest about its tokenizer ceiling, but the 'matches specialists' headline is only true in parts and the unseen-domain rotation claim is undercut by an unmeasured quantization ceiling. read the letter →

arxiv 2506.07886 v3 pith:C6WNMBQN submitted 2025-06-09 cs.CV

Gen Li , Yutong Chen , Yiqian Wu , Kaifeng Zhao , Marc Pollefeys , Siyu Tang This is my paper
classification cs.CV
keywords egocentricvisionmultimodalpretrainingmaskedmodelingcameraposeestimationgazedepthvideosynthesisdiscretetokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a single large masked-pretrained model can serve as a general-purpose engine for egocentric 4D perception and synthesis. The authors introduce EgoM2P, trained on eight egocentric datasets with roughly four billion discrete multimodal tokens, and report that one 400-million-parameter checkpoint predicts gaze, head-camera trajectory, monocular depth, and depth-to-RGB video in a single feed-forward pass with no test-time optimization. The payoff claimed is practical: EgoM2P matches or outperforms task-specific specialist models on all four benchmarks while running an order of magnitude faster, for example predicting a 60-frame camera trajectory in 0.18 seconds. The same checkpoint also transfers to unseen datasets without fine-tuning, which matters for wearable and robotics settings where collecting aligned multimodal annotations is infeasible.

What carries the argument

The load-bearing mechanism is multimodal masked pretraining over discrete spatiotemporal tokens. RGB and depth video are compressed by a frozen Cosmos tokenizer with 8x8x4 spatial-temporal compression into a 64,000-code vocabulary, while gaze dynamics and camera trajectories are tokenized by small trainable VQ-VAE tokenizers with 256-code codebooks and temporal downsampling. A T5-Base encoder-decoder with about 400 million parameters is trained with variable input and target masking using Dirichlet-based token sampling, 3D sine-cosine positional embeddings for video, and modality-specific embedding layers, so any subset of modalities can be used to predict any other. Inference is iterative and parallel, combining classifier-free guidance with nucleus sampling over masked target tokens, which is what allows fast single- or few-step decoding.

What would settle it

Retrain or continue EgoM2P with an egocentric-aware video tokenizer at the same 8x8x4 compression on the same four-billion-token corpus; if EgoExo4D camera RRE does not fall materially below 0.429 and H2O Abs Rel does not improve below 0.055, then the quantization bottleneck the paper identifies is not what limits the model, and the claim that the architecture itself matches specialist accuracy needs re-examination.

Watch

Extended reading notes

Core claim

The paper's central claim is that masked token prediction over heterogeneous egocentric video is enough to learn any-to-any prediction among RGB, depth, gaze, and camera trajectories. On egocentric camera tracking, EgoM2P reports EgoExo4D ATE 0.017, RTE 0.004, and RRE 0.429, comparable to geometry-optimizing specialists such as DROID-SLAM, ACE-Zero, and Align3R while needing 0.18 seconds instead of seconds to minutes, and it generalizes to the unseen ADT dataset. On gaze estimation it reports the lowest MSE among compared methods (0.0162). On monocular depth it reaches 0.055 Abs Rel on H2O and 0.061 on the unseen HOI4D dataset, and on depth-to-RGB synthesis it beats Control-A-Video and ControlVideo on HoloAssist while producing a lower FVD on the unseen ASE dataset. The model achieves these results without geometry modeling or an explicit 3D inductive bias, and missing modalities are handled by masking rather than by pseudo-labeling.

Load-bearing premise

The load-bearing premise is that the frozen Cosmos tokenizer's discrete codebook preserves enough egocentric-specific information for all four tasks; the paper's own Table C.1 shows quantization error already accounts for 0.272 of the 0.429 RRE in EgoExo4D camera tracking, leaving little headroom if the codebook loses fine hand, depth, or head-motion cues.

Editorial extensions

If this is right

  • One 400-million-parameter checkpoint can replace four separate specialist pipelines for gaze, camera tracking, depth, and depth-to-RGB synthesis, with no per-task test-time optimization.
  • Because missing modalities are masked rather than pseudo-labeled, the same pretraining recipe can absorb new egocentric datasets with partial annotation coverage.
  • Zero-shot results on ADT, HOI4D, and ASE suggest the learned token representations transfer across egocentric capture devices, scene types, and synthetic-to-real gaps.
  • Post-training on small amounts of data (roughly 10,000-26,000 samples) lifts the model past the reported baselines, making on-device adaptation to a new scenario cheap.
  • Parallel decoding makes near-real-time egocentric perception plausible: camera trajectory for 60 frames in 0.18 seconds and monocular depth in 0.8 seconds per clip.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: since the paper's ablation shows EgoGen synthetic data improves both tracking and depth, a synthetic generator with reliable gaze labels could let the same masked-pretraining recipe supply gaze supervision to datasets that currently lack it.
  • Editorial inference: the speed numbers are measured with the current frozen video tokenizer, so an egocentric-specific or cheaper tokenizer could widen the real-time margin, though the paper does not study that configuration.
  • Editorial inference: a direct test of temporal awareness would be to shuffle frame order or remove 3D positional embeddings and remeasure camera tracking; if accuracy holds, the video tokenizer's temporal compression may be doing more work than the model's explicit temporal structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces EgoM2P, a unified transformer model pretrained with masked modeling on discrete tokens obtained from RGB video, depth video, gaze dynamics, and camera trajectories in eight egocentric datasets (about four billion tokens). The contributions are a set of temporal tokenizers for gaze and camera trajectories, an extension of 4M-style multimodal masked pretraining to egocentric video with missing modalities, and a unified model that performs gaze estimation, egocentric camera tracking, monocular depth estimation, and conditional video synthesis. The authors report competitive results against specialist baselines on EgoExo4D, H2O, HoloAssist, and unseen-domain evaluations on ADT, HOI4D, and ASE, with substantially faster inference, and they provide ablation studies showing the benefit of synthetic EgoGen data and the effect of tokenization quantization error.

Significance. If the central claims hold, EgoM2P would be a notable step toward a single egocentric foundation model covering perception and synthesis modalities, trained without test-time optimization and with strong efficiency. The paper's strengths include a clean formulation of heterogeneous egocentric data curation, explicit handling of missing modalities, large-scale pretraining with 400M parameters, and an unusually thorough account of tokenizer quantization error in Table C.1. The external benchmarks against independent specialist methods are appropriate, and the paper reports multiple task-level ablations. However, the headline claim that EgoM2P 'matches or outperforms specialist models' is currently supported only partially: several unseen-domain numbers clearly do not match the specialist baselines, and the small evaluation subsets without error bars leave some near-ties unresolved. The missing ADT tokenizer ceiling measurement is a concrete gap that determines how the camera-tracking generalization result should be interpreted.

major comments (4)
  1. [Sec. 4.1 and Sec. 4.2, Tables 2 and 3] The evaluation protocol for camera tracking uses 200 sampled clips and gaze uses 1,000 sampled videos, but no random seeds, confidence intervals, or statistical tests are reported. Several decisive comparisons in Table 2 are very close, such as ATE 0.017 (EgoM2P) versus 0.018 (DROID-SLAM) and RTE 0.004 versus 0.005; with a 200-clip subsample these differences may be within noise. Please report results over multiple seeds or on the full validation split, with confidence intervals or significance tests, before claiming that EgoM2P matches or outperforms the specialist models on these benchmarks.
  2. [Sec. C, Table C.1 and Sec. 4.1] For EgoExo4D camera tracking, the trajectory-tokenizer quantization error is RRE 0.272 while the final EgoM2P RRE is 0.429, meaning the tokenizer accounts for more than half of the total rotation error. The paper's statement that quantization error is 'not the bottleneck' applies to the Euclidean ATE and RTE metrics, but for RRE it is in fact dominant. Without a finer camera-trajectory tokenizer or a continuous pose head, the camera-tracking comparison conflates the model's pose prediction ability with the tokenizer's representation ceiling, and the claim that the model learns camera pose 'without any 3D inductive bias' is not fully supported. Please provide the same quantization-error decomposition for ADT and, ideally, for the other tasks, and discuss how much headroom remains above the tokenizer ceiling.
  3. [Sec. 4.1, Table 2 and Table C.1] The generalization claim on the unseen ADT dataset is not yet established because the ADT trajectory-tokenizer quantization ceiling is not reported. On ADT, EgoM2P achieves RRE 0.490 versus Align3R's 0.276, and post-training improves it only to 0.480. If ADT trajectories quantize as coarsely as EgoExo4D trajectories do, the model may already be near its ceiling and the underperformance is inherited from the tokenizer; if the ADT ceiling is lower, the model has substantial headroom and the underperformance is a genuine model-side limitation. Please measure and report the reconstruction error of the camera-trajectory tokenizer on ADT ground-truth trajectories, and use that value to interpret both the zero-shot and post-trained results.
  4. [Abstract and Sec. 4, Tables 2-4] The headline claim that 'EgoM2P matches or outperforms specialist models' is stronger than the tables support. In Table 2, on ADT the model is worse than Align3R on RRE (0.490 vs 0.276) and ATE (0.032 vs 0.028); in Table 3, on HOI4D it is worse on Abs Rel (0.061 vs 0.045); in Table 4, on ASE it is worse on PSNR (6.923 vs 11.11) and LPIPS (0.715 vs 0.676). Please reword the abstract and conclusion to state that the model is competitive on some benchmarks, stronger on others, and currently underperforms specialist methods on several unseen-domain metrics, with the post-training results clearly separated from zero-shot results.
minor comments (4)
  1. [Sec. 3.5 and Conclusion] There is a terminology inconsistency: the abstract says 'four billion tokens', while Sec. A.3 and the conclusion say '400 billion tokens sampled from our 4 billion token database'. Please clarify that the database contains about 4 billion unique training tokens, from which 400 billion sampled tokens are used for training, or adjust the wording to avoid apparent contradiction.
  2. [Sec. 4.1, paragraph 2] The sentence 'ACE-Zero and Align3R also reply on off-the-shelf monocular depth and optical flow predictions' contains a typo: 'reply' should be 'rely'.
  3. [Table 2] The underlined post-training numbers in the EgoM2P row (0.026, 0.005, 0.480) are not labeled as such in the table caption; please add a footnote or clarify that these correspond to the post-trained model.
  4. [Sec. 3.1 and Sec. 4.3] The depth pseudo-labeling step uses RollingDepth, which is also used as a baseline for depth estimation; please state explicitly whether the RollingDepth pseudo-labels used for training are generated with the same settings as the baseline evaluations, since any overlap or mismatch affects the interpretation of the depth comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: external benchmarks and independent baselines carry the central claims; the EgoGen self-citation is a data-source dependency, not a self-referential derivation.

full rationale

Walking the derivation chain, EgoM2P's central results are evaluated against externally held ground truth and independent third-party baselines. Camera tracking is benchmarked on EgoExo4D and unseen ADT against DROID-SLAM, ACE-Zero, and Align3R (Table 2); depth is benchmarked on H2O and unseen HOI4D against RollingDepth and Align3R (Table 3); gaze is benchmarked against Huang et al. and Lai et al. on EgoExo4D; synthesis is benchmarked against Control-A-Video and ControlVideo on HoloAssist and unseen ASE (Table 4). None of these baselines or metrics are produced by the paper's own pipeline, so the headline comparisons are not reduced to the model's own outputs by construction. The architecture and training scheme inherit T5-Base, the Cosmos tokenizer, and 4M masked modeling from external works, and the modality-specific VQ-VAEs are trained with standard reconstruction losses; the pretraining objective is cross-entropy over discrete tokens, and the evaluation targets are decoded ground-truth-aligned quantities, not the training loss itself. The one prominent self-citation, EgoGen [52], is used as a training-data generator and is ablated in Tables B.2 and B.3; this is an honest data-ingredient analysis, not a claim that EgoM2P's success validates EgoGen, and EgoGen is an independently published simulator with its own external validation. The disclosed quantization errors in Table C.1 are limitations on the achievable ceiling, not evidence of circularity, and the ADT RRE gap versus Align3R is a performance and generalization question, not a logical reduction. No equation in the paper defines a predicted quantity in terms of the evaluation target, and no fitted parameter is renamed as a prediction. The derivation is therefore self-contained with respect to its central multitask claims.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on: (i) a third-party tokenizer's fidelity for egocentric video, (ii) learnability of geometry and gaze from appearance tokens, (iii) quality of RollingDepth pseudo-labels that are also used as a comparison baseline, and (iv) hand-chosen sampling and inference hyperparameters (2048-token cap, Dirichlet concentrations, guidance weight, decoding steps). None of these is itself the target claim, but each can cap the reported performance.

free parameters (4)
  • Maximum visible input/target tokens = 2048
    Chosen via validation-loss ablation (Table B.1: 1024 gives 5.80 loss, 2048 gives 4.93); the central tasks' quality depends on this capacity.
  • Dirichlet concentration parameters for token sampling = 0.01, 0.1, 1, 10
    Picked by hand ('we experiment with different sampling weights... We discover') to balance heavily imbalanced modalities; the paper gives no systematic search.
  • Classifier-free guidance weight omega = Not reported in main text
    Used at inference for all tasks (Sec. 3.4); its value changes prediction sharpness and synthesis quality but is deferred to the supplement.
  • Decoding steps s = 3 to 6 for video; not reported per-task
    Reported only qualitatively in Sec. 3.4 ('increasing decoding steps... is beneficial'); the per-task values behind the benchmark numbers are not given.
assumptions (4)
  • domain assumption Cosmos tokenizer preserves egocentric-relevant information at 8x8x4 compression
    Invoked in Sec. 3.2 as the tokenizer for RGB and depth; the paper's own Table C.1 shows quantization error contributes 0.272 of the 0.429 EgoExo4D RRE, so the downstream ceiling is set by a third-party tokenizer trained mainly on general video.
  • domain assumption Appearance tokens alone determine camera pose and scene geometry without any 3D inductive bias
    Sec. 4.1 claims tracking 'without geometry modeling or 3D inductive bias'; the model must learn rigid-body motion from masked RGB tokens, a strong learnability assumption.
  • domain assumption RollingDepth pseudo-labels are accurate enough to serve as depth training targets
    Sec. 3.1 uses RollingDepth to generate 'pixel-aligned depth videos' for real data; errors in these pseudo-labels bound the achievable depth accuracy and also favor EgoM2P when RollingDepth is the baseline.
  • ad hoc to paper Dataset sampling proportional to dataset size is the optimal balance
    Sec. 3.3 states this was found empirically ('we discover... results in the most stable training'); the choice is validated on the model's own validation loss, not against an external criterion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EgoM2P: Egocentric Multimodal Multitask Pretraining." pith.science (2026). https://pith.science/paper/C6WNMBQN

@misc{pith2026250607886,
  author       = {Pith},
  title        = {Pith review of: EgoM2P: Egocentric Multimodal Multitask Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6WNMBQN}},
  note         = {Machine review of arXiv:2506.07886}
}
read the original abstract

Understanding multimodal signals in egocentric vision, such as RGB video, depth, camera poses, and gaze, is essential for applications in augmented reality, robotics, and human-computer interaction, enabling systems to better interpret the camera wearer's actions, intentions, and surrounding environment. However, building large-scale egocentric multimodal and multitask models presents unique challenges. Egocentric data are inherently heterogeneous, with large variations in modality coverage across devices and settings. Generating pseudo-labels for missing modalities, such as gaze or head-mounted camera trajectories, is often infeasible, making standard supervised learning approaches difficult to scale. Furthermore, dynamic camera motion and the complex temporal and spatial structure of first-person video pose additional challenges for the direct application of existing multimodal foundation models. To address these challenges, we introduce a set of efficient temporal tokenizers and propose EgoM2P, a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding. This unified design supports multitasking across diverse egocentric perception and synthesis tasks, including gaze prediction, egocentric camera tracking, and monocular depth estimation from egocentric video, and also serves as a generative model for conditional egocentric video synthesis. Across these tasks, EgoM2P matches or outperforms specialist models while being an order of magnitude faster. We will fully open-source EgoM2P to support the community and advance egocentric vision research. Project page: https://egom2p.github.io/.

Figures

Figures reproduced from arXiv: 2506.07886 by the authors.

Figure 1
Figure 1. EgoM2P: A large-scale egocentric multimodal and mul￾titask model, pretrained on eight extensive egocentric datasets. It incorporates four modalities—RGB and depth video, gaze dy￾namics, and camera trajectories—to handle challenging tasks like monocular egocentric depth estimation, camera tracking, gaze es￾timation, and conditional egocentric video synthesis. For simplic￾ity, we only visualize four frames here. tric … view at source ↗
Figure 2
Figure 2. Network Architecture: (1) We train VQ-VAE [98] tokenizers for camera trajectories and gaze dynamics (Sec. 3.2), and adopt Cosmos tokenizers [3] to tokenize RGB and depth streams. High-dimensional input modalities, including videos, gaze dynamics, and camera trajectories, are compressed into discrete tokens to serve as our training database. (2) Our EgoM2P follows the architecture of T5-Base [83]. We perform multimod… view at source ↗
Figure 3
Figure 3. Egocentric capture often involves rapid head rota [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Gaze dynamics estimation. EgoM2P can predict results that are more aligned with human intentions. Method H2O [49] HOI4D [60] (unseen) Abs Rel ↓ δ1.25 ↑ Abs Rel ↓ δ1.25 ↑ Time ↓ RollingDepth [44] 0.087 90.5 0.057 97.6 37s Align3R [66] 0.074 91.8 0.045 98.1 90s EgoM2P 0.…
Figure 5
Figure 5. Figure 5: Egocentric video depth estimation. EgoM2P achieves comparable performance with specialist SOTA methods. Rolling Depth [44] struggles in estimating the depth of hands, an important component in egocentric view, while our method can capture hand movement even in the out-…
Figure 6
Figure 6. Figure 6: Comparison of depth-to-RGB video synthesis. Red boxes highlight incorrectly generated fingers in baselines, while ours generate meaningful hand motion. Our results show im￾proved alignment with the input depth, minimizing hallucinations. No post-training was applied fo…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

138 extracted references · 60 canonical work pages

  1. [1]

    Gpt-4v(ision) system card. 2023. 2

    2023

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. Technical Report MSR-TR-2024- 12, Microsoft, 2024. 2

    2024

  3. [3]

    Cosmos world foun- dation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foun- dation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 2, 3, 4, 7, 8, 1

    arXiv 2025

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35: 23716–23736, 2022. 2

    2022

  5. [5]

    Scenescript: Reconstructing scenes with an autoregressive structured language model

    Armen Avetisyan, Christopher Xie, Henry Howard- Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard Newcombe, and Vasileios Balntas. Scenescript: Reconstructing scenes with an autoregressive structured language model. In Euro- pean Conference on Computer Vision (ECC...

    2024

  6. [6]

    Newcombe, and Vasileios Balntas

    Armen Avetisyan, Christopher Xie, Henry Howard- Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard A. Newcombe, and Vasileios Balntas. Scenescript: Reconstructing scenes with an autoregressive structured language model. In ECCV, 2024. 7, 8, 4

    2024

  7. [7]

    MultiMAE: Multi-modal multi-task masked autoencoders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vi- sion, 2022. 5

    2022

  8. [8]

    4M-21: An any-to-any vision model for tens of tasks and modalities

    Roman Bachmann, O ˘guzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Af- shin Dehghan, and Amir Zamir. 4M-21: An any-to-any vision model for tens of tasks and modalities. In Thirty- eighth Conference on Neural Information Processing Sys- tems, 2024. 2, 4, 5, 3

    2024

Show all 138 references
  1. [9]

    Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions

    Sven Bambach, Stefan Lee, David J Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of the IEEE international conference on computer vision , pages 1949–1957, 2015. 3

    1949

  2. [10]

    Introducing hot3d: An egocentric dataset for 3d hand and object tracking.arXiv preprint arXiv:2406.09598,

    Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, et al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking.arXiv preprint arXiv:2406.09598,

    arXiv

  3. [11]

    Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. 1, 2

    2023

  4. [12]

    Align your latents: High-resolution video synthe- sis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthe- sis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1

    2023

  5. [13]

    Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer

    Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cav- allari, ´Aron Monszpart, Daniyar Turmukhambetov, and Victor Adrian Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer. In ECCV, 2024. 6

    2024

  6. [14]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. 2

    2024

  7. [15]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nico- las Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InEuropean conference on computer vision , pages 213–229. Springer,

  8. [16]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. 2

    2021

  9. [17]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models, 2024

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models, 2024. 1

    2024

  10. [18]

    Fleet, and Geoffrey Hinton

    Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. In Advances in Neural Informa- tion Processing Systems, 2022. 2

    2022

  11. [19]

    Control- a-video: Controllable text-to-video diffusion models with motion prior and reward feedback learning, 2024

    Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Ji- ashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control- a-video: Controllable text-to-video diffusion models with motion prior and reward feedback learning, 2024. 7, 8

    2024

  12. [20]

    Scaling egocentric vision: The epic- kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Da- vide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic- kitchens dataset. In European Conference on Computer Vi...

    2018

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, 9 Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arX...

    2010 arXiv

  14. [22]

    Structure and Content-Guided Video Synthesis with Diffusion Mod- els

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and Content-Guided Video Synthesis with Diffusion Mod- els . In 2023 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 7312–7322, Los Alamitos, CA, US...

    2023

  15. [23]

    Black, and Otmar Hilliges

    Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand- object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 3

    2023

  16. [24]

    HOLD: Category-agnostic 3d reconstruction of interacting hands and objects from video

    Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J Black, and Ot- mar Hilliges. HOLD: Category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2024

  17. [25]

    Violet: End-to- end video-language transformers with masked visual-token modeling

    Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to- end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021. 2

    2021 arXiv

  18. [26]

    First-person hand action bench- mark with rgb-d videos and 3d hand pose annotations

    Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action bench- mark with rgb-d videos and 3d hand pose annotations. In Proceedings of Computer Vision and Pattern Recognition (CVPR), 2018. 1

    2018

  19. [27]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023. 2

    2023

  20. [28]

    Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018

    Priya Goyal, Piotr Doll ´ar, Ross Girshick, Pieter Noord- huis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018. 1

    2018

  21. [29]

    Ego4d: Around the World in 3,000 Hours of Egocentric Video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, et al. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Computer Vision and Pattern Recog- nition (CVPR), 2022. 1, 2

    2022

  22. [30]

    Ego-exo4d: Understanding skilled human activity from first- and third-person perspec- tives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, et al. Ego-exo4d: Understanding skilled human activity from first- and third-person perspec- tives. 2024 IEEE/CVF Conference on Computer Visio...

    2024

  23. [31]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. In The Twelfth International Conference on Learning Representa- tio...

    2024

  24. [32]

    World models

    David Ha and J ¨urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 2

    2018 arXiv

  25. [33]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll’ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15979–15988, 2021. 5

    2022

  26. [34]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 5

    2021

  27. [35]

    Video diffu- sion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffu- sion models. In Advances in Neural Information Processing Systems, pages 8633–8646. Curran Associates, Inc., 2022. 1

    2022

  28. [36]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In In- ternational Conference on Learning Representations, 2020. 5

    2020

  29. [37]

    Cogvideo: Large-scale pretraining for text-to- video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to- video generation via transformers. In The Eleventh Inter- national Conference on Learning Representations, 2023. 1

    2023

  30. [38]

    Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans

    Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Au- toregressive diffusion models. In International Conference on Learning Representations, 2022. 5

    2022

  31. [39]

    Ross, and Alireza Fathi

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual- language pre-training with multi-source multimodal knowl- edge memory. 2023 IEEE/CVF Conference on Computer Vision and...

    2023

  32. [40]

    Predicting gaze in egocentric video by learning task- dependent attention transition

    Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning task- dependent attention transition. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV , pages 789–

    2018

  33. [41]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2

    2024 arXiv

  34. [42]

    Epic-fusion: Audio-visual temporal binding for egocentric action recognition

    Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 5492–5501, 2019. 3

    2019

  35. [43]

    Re- purposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Re- purposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR),

  36. [44]

    Video depth without video models

    Bingxin Ke, Dominik Narnhofer, Shengyu Huang, Lei Ke, Torben Peters, Katerina Fragkiadaki, Anton Obukhov, and 10 Konrad Schindler. Video depth without video models. In CVPR, 2025. 3, 7

    2025

  37. [45]

    Text2Video-Zero: Text- to-Image Diffusion Models are Zero-Shot Video Generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2Video-Zero: Text- to-Image Diffusion Models are Zero-Shot Video Generators . In 2023 IEEE/CVF International Conference on Com- puter Vision (...

    2023

  38. [46]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 2

    2023

  39. [47]

    Harmsen, and Neil Houlsby

    Alexander Kolesnikov, Andr ´e Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah J. Harmsen, and Neil Houlsby. UVim: A unified modeling approach for vision with learned guiding codes. In Advances in Neural Information Processing Systems, 2022. 2

    2022

  40. [48]

    VideoPoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jose Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, et al. VideoPoet: A large language model for zero-shot video generation. In Proceedings of the 41st International Con- ference on Machine Learning, pages 25105–25124. PMLR,

  41. [49]

    H2o: Two hands manipulating objects for first person interaction recognition

    Taein Kwon, Bugra Tekin, Jan St ¨uhmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 10138–10148, 2021. 1, 3, 5, 7, 4

    2021

  42. [50]

    Bolin Lai, Miao Liu, Fiona Ryan, and James M. Rehg. In the eye of transformer: Global-local correlation for egocen- tric gaze estimation. In 33rd British Machine Vision Con- ference 2022, BMVC 2022, London, UK, November 21-24, 2022, page 227. BMV A Press, 2022. 6

    2022

  43. [51]

    Lisa: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. In 2024 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 9579–9589, 2024. 1

    2024

  44. [52]

    Egogen: An egocentric synthetic data generator

    Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mi- hai Dusmanu, Yan Zhang, Marc Pollefeys, and Siyu Tang. Egogen: An egocentric synthetic data generator. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 14497–14509,

  45. [53]

    Videochat: Chat-centric video understanding

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2

    2023 arXiv

  46. [54]

    Megasam: Accurate, fast and robust structure and motion from casual dynamic videos

    Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust structure and motion from casual dynamic videos. In Proceedings of the Computer Vision and Pattern Recogn...

    2025

  47. [55]

    Video-LLaV A: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual representation by alignment before projection. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 5971–5984, Miami, Fl...

    2024

  48. [56]

    Cross-view exocentric to egocentric video synthesis

    Gaowen Liu, Hao Tang, Hugo M Latapie, Jason J Corso, and Yan Yan. Cross-view exocentric to egocentric video synthesis. In Proceedings of the 29th ACM International Conference on Multimedia, pages 974–982, 2021. 3

    2021

  49. [57]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2

    2024

  50. [58]

    Exocentric-to-egocentric video gener- ation

    Jia-Wei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou. Exocentric-to-egocentric video gener- ation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 3

    2024

  51. [59]

    Li, Ying Shan, and Ge Li

    Ruyang Liu, Chen Li, Yixiao Ge, Thomas H. Li, Ying Shan, and Ge Li. BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13658–13667, Los Alamitos, CA, USA,

    2024

  52. [60]

    Hoi4d: A 4d egocentric dataset for category-level human-object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...

    2022

  53. [61]

    Y . Liu, H. Yang, X. Si, L. Liu, Z. Li, Y . Zhang, Y . Liu, and L. Yi. Taco: Benchmarking generalizable bimanual tool- action-object understanding. In 2024 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 21740–21751, Los Alamitos, CA, USA, 2024. ...

    2024

  54. [62]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Fe- ichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976– 11986, 2022. 2

    2022

  55. [63]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learn- ing Representations, 2019. 1, 2

    2019

  56. [64]

    Unified-io 2: Scaling autoregressive mul- timodal models with vision, language, audio, and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Anirud- dha Kembhavi. Unified-io 2: Scaling autoregressive mul- timodal models with vision, language, audio, and action. 2024 IEEE/CVF Conference on Computer Vision and Pat- ter...

    2024

  57. [65]

    UNIFIED-IO: A uni- fied model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A uni- fied model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Repre- sentations, 2023. 2

    2023

  58. [66]

    Align3r: Aligned monocular depth estimation for dynamic videos

    Jiahao Lu, Tianyu Huang, Peng Li, Zhiyang Dou, Cheng Lin, Zhiming Cui, Zhen Dong, Sai-Kit Yeung, Wenping Wang, and Yuan Liu. Align3r: Aligned monocular depth estimation for dynamic videos. In CVPR, 2025. 6, 7 11

    2025

  59. [67]

    Dream Machine

    LumaLabs. Dream Machine. https://lumalabs. ai/dream-machine, 2024. 1

    2024

  60. [68]

    Videofusion: Decomposed diffusion models for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liangsheng Wang, Yujun Shen, Deli Zhao, Jinren Zhou, and Tien-Ping Tan. Videofusion: Decomposed diffusion models for high-quality video generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), ...

    2023

  61. [69]

    Aria ev- eryday activities dataset

    Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexan- der Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria ev- eryday activities dataset. arXiv preprint arXiv:2402.13349,

    arXiv

  62. [70]

    Nymeria: A massive collection of multimodal egocentric daily motion in the wild

    Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. arXiv preprint arXiv:2406.09905, 2024. 1

    2024 arXiv

  63. [71]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (ACL 2024) , 2024. 1, 2

    2024

  64. [72]

    Mm1: Methods, analysis & insights from multimodal llm pre-training, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier Biard, Sam Dodge, Philipp Dufter, Bowen Zhang, Dhruti Shah, Xianzhi Du, Futang Peng, Haotian Zhang, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training, 2024. 2

    2024

  65. [73]

    Project Aria Glasses

    Meta. Project Aria Glasses. https : / / www . projectaria.com/, 2023. 3

    2023

  66. [74]

    Trans- formers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and Franc ¸ois Fleuret. Trans- formers are sample-efficient world models. arXiv preprint arXiv:2209.00588, 2022. 2

    2022 arXiv

  67. [75]

    HoloLens 2

    Microsoft. HoloLens 2. https://www.microsoft. com/en-us/hololens, 2019. 3, 1

    2019

  68. [76]

    4M: Massively multimodal masked modeling

    David Mizrahi, Roman Bachmann, O ˘guzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Za- mir. 4M: Massively multimodal masked modeling. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023. 2, 4, 5, 3

    2023

  69. [77]

    AssemblyHands: towards egocentric activity understanding via 3d hand pose esti- mation

    Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. AssemblyHands: towards egocentric activity understanding via 3d hand pose esti- mation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12999–1300...

    2023

  70. [78]

    Video generation models as world simula- tors

    OpenAI. Video generation models as world simula- tors. https : / / openai . com / index / video - generation- models- as- world- simulators/,

  71. [79]

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El- Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Mich...

    2024

  72. [80]

    Aria digital twin: A new benchmark dataset for egocentric 3d machine perception

    Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Pe- ters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Carl Yuheng Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In ICCV, 2023. 6, 3, 4

    2023

  73. [81]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjan- dra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024. 1, 2

    arXiv

  74. [82]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International conference on machine learning...

    2021

  75. [83]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.J. Mach. Learn. Res., 21(1), 2020. 4, 5

    2020

  76. [84]

    High-Resolution Im- age Synthesis with Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-Resolution Im- age Synthesis with Latent Diffusion Models . In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, Los Alamitos, CA, USA, 2022. IEE...

    2022

  77. [85]

    Gen-3 Alpha

    RunwayML. Gen-3 Alpha. https://runwayml.com/ research/introducing-gen-3-alpha , 2024. 1

    2024

  78. [86]

    Lamar: Bench- marking localization and mapping for augmented reality

    Paul-Edouard Sarlin, Mihai Dusmanu, Johannes L Sch¨onberger, Pablo Speciale, Lukas Gruber, Viktor Lars- son, Ondrej Miksik, and Marc Pollefeys. Lamar: Bench- marking localization and mapping for augmented reality. In European Conference on Computer Vision, pages 686–704. Sprin...

    2022

  79. [87]

    Sener, D

    F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly101: A large-scale multi- view video dataset for understanding procedural activities. CVPR 2022. 1

    2022

  80. [88]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In The Eleventh International Conference on Lea...

    2023

  81. [89]

    Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur- Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler G...

    1906 arXiv

  82. [90]

    Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, and Oliver Lemon

    Alessandro Suglia, Claudio Greco, Katie Baker, Jose L. Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, and Oliver Lemon. AlanaVLM: A multimodal embodied AI foundation model for egocentric video understanding. In Findings of the Association for Computational Linguis- ...

    2024

  83. [91]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2024. 2

    2024

  84. [92]

    Gem- ini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gem- ini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  85. [93]

    Kling ai video generator

    Kuaishou Technology. Kling ai video generator. https: //kling.kuaishou.com/en, 2024. Accessed: 2025- 03-03. 2

    2024

  86. [94]

    DROID-SLAM: Deep vi- sual SLAM for monocular, stereo, and RGB-d cameras

    Zachary Teed and Jia Deng. DROID-SLAM: Deep vi- sual SLAM for monocular, stereo, and RGB-d cameras. In NeuRIPS, 2021. 3, 6

    2021

  87. [95]

    VideoMAE: Masked autoencoders are data-efficient learn- ers for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learn- ers for self-supervised video pre-training. In Advances in Neural Information Processing Systems, 2022. 2, 5

    2022

  88. [96]

    Towards accurate generative models of video: A new metric & challenges, 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges, 2019. 7

    2019

  89. [97]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 2

    2024 arXiv

  90. [98]

    Neural discrete representation learn- ing

    A ¨aron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Neural Information Processing Systems , 2017. 3, 4

    2017

  91. [99]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in Neural Information Processing Systems . Curran Associates, Inc., 2017. 2

    2017

  92. [100]

    Phenaki: Variable length video generation from open do- main textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open do- main textual descriptions. In International Conference on ...

    2023

  93. [101]

    Ego-only: Egocentric action detection without exocentric transferring

    Huiyu Wang, Mitesh Kumar Singh, and Lorenzo Torresani. Ego-only: Egocentric action detection without exocentric transferring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5250–5261, 2023. 3

    2023

  94. [102]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 14549–14560,

  95. [103]

    OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learn- ing framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learn- ing framework. In International Conference on Machine Learni...

    2022

  96. [104]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024. 7

    2024

  97. [105]

    Image as a foreign language: BEiT pretraining for vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. InProceedings of the IEEE/CVF Con...

  98. [106]

    Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world

    Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In...

    2023

  99. [107]

    Videocomposer: Compositional video syn- thesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video syn- thesis with motion controllability. In Thirty-seventh Con- ference on Neural Information Processing Systems , 2023. 1

    2023

  100. [108]

    Egovid-5m: A large-scale video-action dataset for egocentric video generation

    Xiaofeng Wang, Kang Zhao, Feng Liu, Jiayu Wang, Gu- osheng Zhao, Xiaoyi Bao, Zheng Zhu, Yingya Zhang, and Xingang Wang. Egovid-5m: A large-scale video-action dataset for egocentric video generation. arXiv preprint arXiv:2411.08380, 2024. 3

    2024 arXiv

  101. [109]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative lear...

    2022 arXiv

  102. [110]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yan- song Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2: Scaling foundation models for multimodal vide...

    2024

  103. [111]

    Tune-a-video: One-shot tuning 13 of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning 13 of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Confere...

    2023

  104. [112]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat- Seng Chua. Next-gpt: Any-to-any multimodal llm. In Proceedings of the International Conference on Machine Learning, pages 53366–53397, 2024. 1

    2024

  105. [113]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024

  106. [114]

    VideoCLIP: Contrastive pre- training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre- training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural ...

    2021

  107. [115]

    X- gen: Ego-centric video prediction by watching exo-centric videos

    Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. X- gen: Ego-centric video prediction by watching exo-centric videos. In The Thirteenth International Conference on Learning Representations, 2025. 3

    2025

  108. [116]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. In CVPR,

  109. [117]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yux- uan.Zhang, Weihan Wang, Yean Cheng, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an ...

    2025

  110. [118]

    MMEgo: To- wards building egocentric multimodal LLMs

    Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Haoxuan You, Dan Xu, Zhe Gan, Jiasen Lu, Yinfei Yang, and Bowen Zhang. MMEgo: To- wards building egocentric multimodal LLMs. In The Thir- teenth International Conference on Learning Representa- tions, 2025. 2

    2025

  111. [119]

    Diffusion-guided reconstruction of every- day hand-object interaction clips

    Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shub- ham Tulsiani. Diffusion-guided reconstruction of every- day hand-object interaction clips. In Proceedings of the IEEE/CVF international conference on computer vision , pages 19717–19728, 2023. 3

    2023

  112. [120]

    Vector-quantized image modeling with improved VQGAN

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In International Conference on Learn- ing Representations, 2022. 4, 1

    2022

  113. [121]

    When and why vision- language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. 1

    2023

  114. [122]

    OakInk2 : A Dataset of Bimanual Hands-Object Manipulation in Com- plex Task Completion

    Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Han- lin Xu, Zenan Lin, Kailin Li, and Cewu Lu. OakInk2 : A Dataset of Bimanual Hands-Object Manipulation in Com- plex Task Completion . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 445–4...

    2024

  115. [123]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Conference on Empirical Methods in Natural Language Processing, 2023. 1

    2023

  116. [124]

    Efros, Eli Shecht- man, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2018. 7

    2018

  117. [125]

    Egobody: Human body shape and motion of interacting people from head-mounted devices

    Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In European Confer- ence on Computer Vision, pages 180–200. Springer, 2022. 1

    2022

  118. [126]

    Controlvideo: Training-free controllable text-to-video generation

    Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Revi...

    2024

  119. [127]

    Learning video representations from large lan- guage models

    Yue Zhao, Ishan Misra, Philipp Kr ¨ahenb¨uhl, and Rohit Girdhar. Learning video representations from large lan- guage models. In CVPR, 2023. 1

    2023

  120. [128]

    Gimo: Gaze-informed human motion prediction in context

    Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIII...

    2022

  121. [129]

    Temporal perception and prediction in ego-centric video

    Yipin Zhou and Tamara L Berg. Temporal perception and prediction in ego-centric video. In Proceedings of the IEEE International Conference on Computer Vision, pages 4498– 4506, 2015. 3

    2015

  122. [130]

    Batch size

    Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. On the continuity of rotation representations in neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3 14 EgoM2P: Egocentric Multimodal Multitask Pretraining – Supplement...

    2019

  123. [133]

    We select αi with uniform probability: α1 = (0.01, 0.01, 0.01, 0.01), α2 = (0.1, 0.1, 0.1, 0.1), α3 = (1, 1, 1, 1), α4 = (10, 10, 10, 10)

    Sampling a Dirichlet Distribution: Let α1, α2, α3, α4 be the four concentration parameters of the Dirichlet dis- 1 tributions. We select αi with uniform probability: α1 = (0.01, 0.01, 0.01, 0.01), α2 = (0.1, 0.1, 0.1, 0.1), α3 = (1, 1, 1, 1), α4 = (10, 10, 10, 10). If the data...

  124. [134]

    Here, θ represents a probability distribution over available modalities

    Sampling a Probability Vector: When αi is sam- pled, we then sample a probability vectorθ from the chosen Dirichlet distribution: θ ∼ Dirichlet(αi). Here, θ represents a probability distribution over available modalities

  125. [135]

    , θn), the number of tokens for each modality i is Ti = 2048× θi

    Sampling Tokens from the Modalities: Finally, given θ = (θ1, θ2, . . . , θn), the number of tokens for each modality i is Ti = 2048× θi. Within this cap, tokens from each modality are sampled randomly. 2048 is the maximum number of input and target tokens. A.3. EgoM2P Pretrain...

  126. [136]

    Sampling datasets based on probabilities proportional to their sizes

  127. [137]

    Sampling datasets using a uniform distribution

  128. [138]

    Sampling datasets with probabilities proportional to the logarithm of their sizes. We observe that across all datasets in our database, the first sampling method, which selects datasets based on probabilities proportional to their sizes, consistently results in the lowest vali...

  129. [256]

    Camera Traj

    The number of parameters for gaze and camera tok- Configuration Gaze Dyn. Camera Traj. Codebook size 256 Temporal compression 2 Code latent dimension 32 EMA dead code threshold 2 Codebook EMA 0.99 l2-normalized codes [120] ✓ Codebook weight 1.0 Commitment weight β 1.0 Encoder ...

  130. [2024]

    IEEE Computer Society. 1

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.