Pith. sign in

REVIEW 1 major objections 7 minor 88 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

3D-grounded language model decodes motion and description together

2026-07-09 21:52 UTC pith:Z3C4U7HV

load-bearing objection Solid egocentric motion forecasting with 3D scene grounding, but the GRPO cross-modal alignment gains are partly circular and need untangling. the 1 major comments →

arxiv 2607.07001 v1 pith:Z3C4U7HV submitted 2026-07-08 cs.CV

Ego-Human Motion Prediction with 3D-Aware LLM

Yujin Bae , Jaewoo Jeong , Hyeonseong Kim , Kuk-Jin Yoon This is my paper
classification cs.CV
keywords motionego3dlmposepredictionsemanticegocentricfuturelanguage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that predicting what a person will do next from a head-mounted camera is fundamentally ill-posed unless the model understands the surrounding 3D space and reasons about motion and language as a single coupled output. The authors build Ego3DLM, a language model that takes sparse head-and-hand tracking, egocentric video, and a precomputed 3D scene point cloud, and in one autoregressive pass simultaneously produces four things: the reconstructed past body pose, the predicted future body pose, a natural-language narration of what just happened, and a description of what will happen next. The central claim is that generating all four outputs in a single token stream forces the motion and the words to stay mutually consistent — the description of the future must match the pose the model actually predicts, and the pose must respect the obstacles and affordances the scene reasoning identified. A three-stage training pipeline builds this capability: first the model learns to answer questions about objects and navigable space in 3D scenes, then it is instruction-tuned to produce all four outputs with a spatial-reasoning preamble, and finally a reinforcement-learning stage (GRPO) uses rewards that explicitly measure whether the predicted motion and the predicted text actually correspond to each other. On the Nymeria egocentric benchmark the model improves over prior methods on every motion and language metric, with the largest relative gains on the cross-modal alignment distance between predicted motion and predicted text.

Core claim

When a language model is trained to generate past pose, future pose, past narration, and future narration in a single autoregressive sequence — prepended by an explicit spatial-reasoning step over 3D scene features — the four outputs become more accurate than when any subset is predicted independently, and the simultaneously generated motion and text align more closely in a shared embedding space than outputs produced by separate decoders or by methods that inject scene features without explicit scene reasoning. The inter-modal reward in the GRPO stage, which penalizes inconsistency between predicted motion and predicted text, produces reciprocal gains: motion accuracy and text quality both,

What carries the argument

Three components carry the argument. First, a 3D scene feature extractor lifts 2D semantic features from egocentric video onto a 3D point cloud, voxelizes it, and compresses it via a Q-Former into 32 query embeddings injected into the language model. Second, a spatial-semantic pretraining stage trains the model on automatically generated QA pairs about object identity, placement, directional clearance, and collision risk before any motion task is introduced. Third, a GRPO reinforcement stage optimizes an inter-modal matching reward R_matching = -(d_gp + d_pg + d_pp), where each term is the Euclidean distance in a shared embedding space between a text embedding and a motion embedding — d_gp (

Load-bearing premise

The model assumes a precomputed 3D scene feature is available at inference time. The ablation shows that removing 3D scene input causes the most severe degradation across all tasks, and the entire spatial-semantic pretraining and spatial-reasoning pipeline depends on this input existing. For novel or outdoor environments where a pre-built 3D map is unavailable, the gains that distinguish this model from baselines would likely shrink or vanish.

What would settle it

If one removed the single-pass joint decoding and instead generated each of the four outputs independently using the same model and same 3D scene features, and the cross-modal alignment distance d_pp did not worsen relative to joint decoding, the core claim that holistic single-pass generation is necessary for cross-modal consistency would be undermined. The paper's own Table B provides partial evidence against this falsifier (joint decoding reduces d_pp by 9.25% vs. separate), but a stronger test would vary the decoding order and the number of outputs generated jointly.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • AR/VR systems that can maintain a running 3D scene map could use this architecture to generate both what the user will do and a natural-language explanation of why, enabling proactive assistance that is both physically grounded and interpretable.
  • The inter-modal reward mechanism is modality-agnostic in principle: any pair of modalities that can be embedded in a shared space could benefit from a matching reward during reinforcement finetuning, suggesting applicability beyond pose-language pairs.
  • The finding that joint past-future decoding improves tracking more than forecasting (28% APE reduction in tracking vs. 7% JPE reduction in forecasting) suggests that temporal coupling is a stronger inductive bias for reconstruction than for prediction, which has implications for how multi-task motion models are designed.
  • The spatial-reasoning preamble acting as a chain-of-thought prompt that propagates into downstream motion and language outputs raises the question of whether other structured reasoning steps (e.g., social context, object affordance, temporal causality) prepended to generation would similarly improve grounded prediction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The paper introduces Ego3DLM, a framework for egocentric human motion prediction that simultaneously decodes past pose, future pose, past narration, and future narration in a single autoregressive pass, conditioned on three-point tracking, egocentric video, and 3D scene features. The method employs a three-stage training scheme: (1) spatial-semantic scene awareness pretraining via auto-generated QA pairs, (2) holistic multi-task instruction tuning with a prepended spatial scene reasoning step, and (3) GRPO-based reinforcement finetuning with intra- and inter-modal rewards. Experiments on the Nymeria benchmark show state-of-the-art performance across motion prediction, tracking, and text description metrics, with ablations isolating each training stage, input modality, and reward component.

Significance. The paper addresses a well-motivated problem: egocentric motion forecasting is inherently ill-posed, and prior work either neglects 3D scene context or treats pose and language as separate streams. The simultaneous four-output decoding design is a genuine architectural contribution, and the three-stage training pipeline is well-structured. The ablation in Table 3 showing that removing 3D scene input causes the most severe degradation is a clear and important finding. The spatial-semantic QA dataset construction (Sec. 3.3, Supp. C5) is a useful contribution. The project page is publicly available, which supports reproducibility. However, the significance of the GRPO stage is partially undermined by a train-on-test overlap in the cross-modal alignment metrics, as detailed below.

major comments (1)
  1. §3.5, Eqs. (5)–(6) vs. Table 1 and Table 4: The inter-modal reward R_matching (Eq. 6) directly includes d_pp as one of its three terms, and R_text (Eq. 5) uses BLEU-4. The evaluation in Table 1 then reports d_pp and Bleu-4 as primary evidence of cross-modal coherence improvement. This creates a direct train-on-test overlap: the GRPO stage explicitly optimizes the exact quantities later cited as evaluation metrics. Table 4 confirms the concern — the largest GRPO gain is on d_pp (5.9170 → 4.2571, a 28% reduction), while independently-measured metrics show marginal gains (Bleu-4: 0.1036 → 0.1039, a 0.3% improvement; APE: 148.5 → 147.9, a 0.4% improvement). The disproportionate improvement on the directly-optimized metric versus near-stagnation on independent metrics is consistent with the model exploiting the embedding space rather than achieving genuine cross-modal coherence. This is load-
minor comments (7)
  1. §3.2: The Q-Former compresses the scene into K=32 query embeddings. A brief sensitivity analysis or justification for this choice would strengthen the paper.
  2. Table 1: The 'LLM (Qwen 2.5 7B)' baseline is described as zero-shot in Supp. C7 but not clearly labeled as such in the table caption.
  3. §4.1: The data split uses a 4:1 train/validation ratio at the scene level. The exact scene IDs or a released split specification would aid reproducibility; the authors state this will be released with code.
  4. Supp. C8: All embedding-based metrics (FID, Diversity, d_gp, d_pg, d_pp, R@N) use encoders trained on the same training split of Nymeria. A note acknowledging potential split-specific bias would be appropriate.
  5. Fig. 2 caption (corrected in Supp. B): The original caption contained an error describing the pretraining process. The correction should be incorporated into the main text figure caption.
  6. §3.1: The PQ-VAE uses two codebooks (Supp. C1), but the main text mentions C=4096 codebooks. This should be clarified — the vocabulary is 4096 entries per codebook, with two codebooks total.
  7. Table 3: The 'w/o SSR' ablation shows modest degradation, but the spatial scene reasoning step's contribution is not analyzed in depth. A qualitative example of SSR output would help readers understand its role.

Circularity Check

2 steps flagged

GRPO reward function directly includes the same metrics (d_pp, BLEU-4, JPE) later reported as evidence of improvement, creating a fitted-input-called-prediction pattern for the GRPO stage; however, the broader framework has independent support from non-optimized metrics and the instruction-tuning stage.

specific steps
  1. fitted input called prediction [Eq. 5-6 and Table 4 (GRPO ablation)]
    "r = w_m · max(0, 1−JPE) [R_motion] + w_t · BLEU [R_text] + w_d · R_matching + R_format ... R_matching = −[d_gp(e_t^gt, e_m^pred) + d_pg(e_t^pred, e_m^gt) + d_pp(e_t^pred, e_m^pred)]"

    The GRPO reward (Eq. 5-6) directly includes JPE, BLEU-4, and d_pp as optimization targets. Table 4 then reports these exact same metrics as evidence that GRPO improves the model. The largest GRPO gain is on d_pp (5.9170→4.2571, 28% reduction)—the directly optimized quantity—while independently-measured metrics show marginal gains (Bleu-4: 0.3%, APE: 0.4%). The paper states: 'incorporating the inter-modal matching reward R_matching yields broad improvements across both motion and text metrics simultaneously, including motion prediction, narration Bleu scores, and motion-description alignment distances.' But the improvement on d_pp is tautological: the model is explicitly rewarded for minimizing it, then the reduction is cited as evidence of 'comprehensive and semantically grounded cross-mod

  2. fitted input called prediction [Table 1, d_pp column and Sec. 3.5]
    "R_matching = −[d_gp(e_t^gt, e_m^pred) + d_pg(e_t^pred, e_m^gt) + d_pp(e_t^pred, e_m^pred)] ... Most strikingly, the motion-description embedding distance (x-y Align.) improves by 56.8%, indicating that our simultaneously predicted motion and descriptions are tightly aligned in a shared semantic space."

    The d_pp metric reported in Table 1 (4.2571 for Ego3DLM vs. 9.8686 for EgoLM) is the same d_pp that appears in the GRPO reward R_matching (Eq. 6). The 56.8% improvement cited as evidence that 'holistic joint prediction yields semantically coherent motion-language outputs' is partly a direct consequence of optimizing that exact quantity during training. EgoLM does not use GRPO and thus does not optimize d_pp, making the comparison on this specific metric asymmetric. The claim of 'semantically coherent' motion forecasts rests substantially on this directly-optimized metric.

full rationale

The paper's core architectural contribution—holistic single-pass decoding of past/future pose and text with 3D scene grounding—is supported by independent evidence: the instruction-tuning ablation (Table 3) shows gains on metrics not in any reward function (APE, FDE, RougeL, SBert), and the comparison against baselines on standard motion metrics (APE, ADE, FDE) that are not GRPO-optimized still favors Ego3DLM. The circularity is specifically in the GRPO stage (Stage III): the reward function (Eq. 5-6) directly includes JPE, BLEU-4, and d_pp, and Table 4 reports these same metrics as evidence of GRPO's effectiveness. The most clear-cut case is d_pp, which shows the largest GRPO improvement (28%) and is simultaneously the most directly optimized term in R_matching. This is a fitted-input-called-prediction pattern: the model is trained to minimize d_pp, then the reduced d_pp is presented as evidence of 'cross-modal coherence.' The encoders used for d_pp are from external work (Guo et al. [26], not self-cited) and trained on the authors' training split, which is standard practice but means the embedding space is split-specific. The circularity is partial rather than total: non-optimized metrics also improve, and the instruction-tuning results provide independent support for the holistic decoding claim. Score 4 reflects that the GRPO-specific evidence is partly circular while the broader framework retains independent content.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 2 invented entities

The free parameters are mostly standard RL/architectural hyperparameters with reasonable values. The key axioms are the 3D scene availability assumption (acknowledged as a limitation) and the reliance on auto-generated QA data without human validation. The invented entities are methodological constructs rather than physical entities; the QA dataset has independent falsifiable metrics but the matching reward's validity is partially circular.

free parameters (8)
  • w_m (motion reward weight) = 1.0
    Set by hand for GRPO reward; controls relative influence of motion accuracy reward
  • w_t (text reward weight) = 0.8
    Set by hand for GRPO reward; controls relative influence of text quality reward
  • w_d (matching reward weight) = 0.02
    Set by hand for GRPO reward; controls relative influence of inter-modal consistency reward
  • K (Q-Former query count) = 32
    Chosen architectural parameter for scene embedding compression
  • C (PQ-VAE codebook size) = 4096
    Vocabulary extension size for motion tokens; follows EgoLM
  • G (GRPO group size) = 6
    Number of candidate outputs sampled per prompt in GRPO
  • epsilon (GRPO clipping) = 0.2
    PPO-style clipping parameter
  • beta (KL coefficient) = 0.001
    Controls KL divergence penalty in GRPO objective
axioms (5)
  • domain assumption 3D scene features are available at inference time
    Sec. 5 Limitations explicitly states 'we assume the 3D scene feature is given.' This is required for the spatial reasoning step and all scene-conditioned predictions.
  • domain assumption Motion can be faithfully represented as discrete tokens via PQ-VAE
    Sec. 3.1: pose is tokenized via PQ-VAE with 4096 codebooks. The quality of this discretization bounds achievable motion accuracy. Follows EgoLM.
  • ad hoc to paper Auto-generated QA pairs from Qwen2.5-VL-7B provide reliable scene understanding supervision
    Sec. 3.3: semantic awareness QA pairs are generated by Qwen2.5-VL-7B without human validation. The pretraining stage depends on these being accurate.
  • domain assumption Joint autoregressive generation of pose and text tokens in a single sequence improves cross-modal consistency
    Sec. 3.4: the instruction tuning objective assumes that generating all four outputs in one pass enforces consistency. This is the core design principle; Table B ablation provides partial evidence.
  • ad hoc to paper GRPO with learned-encoder-based rewards can optimize cross-modal alignment
    Sec. 3.5: the inter-modal reward R_matching uses encoders trained on the same data split. The assumption that this reward signal generalizes beyond the encoder's training distribution is unstated.
invented entities (2)
  • Spatial-semantic scene awareness QA dataset independent evidence
    purpose: Pretraining the LM to understand 3D scene layout and object semantics before motion prediction
    The dataset is described in detail (535K spatial QA, 115K semantic QA, 208 scenes) with generation algorithm (Algorithm 1). Collision/freespace metrics provide falsifiable accuracy measures (Table 2: 0.7764 collision, 0.8848 freespace).
  • Inter-modal matching reward R_matching no independent evidence
    purpose: GRPO reward signal measuring pose-language consistency via three embedding distance terms (d_gp, d_pg, d_pp)
    The reward depends on modality-specific encoders trained on the authors' own data split. No external benchmark validates that this reward measures true cross-modal alignment rather than distribution-specific embedding proximity.

pith-pipeline@v1.1.0-glm · 30535 in / 3833 out tokens · 287188 ms · 2026-07-09T21:52:32.078925+00:00 · methodology

0 comments
read the original abstract

Anticipating human motion from an egocentric perspective is fundamental for proactive assistance in AR/VR, human-robot collaboration, and embodied AI. While recent works incorporate language as a semantic prior to reduce the ill-posed nature of egocentric forecasting, they largely neglect the 3D spatial and semantic context that governs how motion unfolds, and treat pose and language prediction as separate inference streams. We introduce Ego3DLM, built on two core principles: accurate motion forecasting requires explicit spatial and semantic understanding of the 3D environment, and pose and language must be predicted holistically in a single pass, since motion is inherently tied to the semantic interpretation of actions being performed. Given three-point tracking, 3D scene features, and egocentric video, Ego3DLM simultaneously decodes past pose, future pose, past narration, and future narration in a single autoregressive pass, grounding predicted poses and descriptions in one another to enforce cross-modal and temporal consistency. We adopt a three-stage training scheme: (1) spatial-semantic scene awareness pretraining; (2) holistic instruction tuning over all four outputs in a single pass; and (3) GRPO-based reinforcement finetuning with intra- and inter-modal rewards that directly optimize pose-language fidelity. Experiments on the Nymeria benchmark demonstrate that Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description, showing that 3D scene grounding and holistic cross-modal prediction yield physically plausible and semantically coherent motion forecasts. The project page is available at https://jaewoo97.github.io/Ego3DLM/.

Figures

Figures reproduced from arXiv: 2607.07001 by Hyeonseong Kim, Jaewoo Jeong, Kuk-Jin Yoon, Yujin Bae.

Figure 1
Figure 1. Figure 1: Our Ego3DLM incorporates the semantic context of the surrounding 3D envi￾ronment along with 2D egocentric video and three-point tracking data to generate past and future poses and their corresponding language motion descriptions. near-field interactions that matter for planning [23, 24, 41]. Yet forecasting from this viewpoint is inherently ill-posed: the camera captures little of the body, only sparse wea… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Ego3DLM framework. Our model consists of three stages. 3D Scene Feature Extraction (Sec. 3.2): 2D semantic features are extracted from egocentric video frames and lifted onto the 3D point cloud. The feature-enhanced point cloud is fed into a Q-Former to produce compact scene query embeddings (top left). LM Training: (I) Pre-training (Sec. 3.3) aligns scene, motion, and language via two comp… view at source ↗
Figure 3
Figure 3. Figure 3: Spatial-semantic scene awareness QA dataset generation. Semantic [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results for motion forecasting and tracking. The ground-truth (GT) past motion trajectory is shown in red, and the predicted future motion is visualized in blue. For future motion narration below the figure, green highlights correct phrases matching the GT, while red highlights incorrect or mismatched descriptions. 4.3 Qualitative Prediction [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

88 extracted references · 88 canonical work pages · 13 internal anchors

  1. [1]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Ashutosh, K., Pavlakos, G., Grauman, K.: Fiction: 4d future interaction prediction from video. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 17613–17625 (2025)

    work page 2025

  2. [2]

    In: 2022 International Conference on 3D Vision (3DV)

    Athanasiou, N., Petrovich, M., Black, M.J., Varol, G.: Teach: Temporal action composition for 3d humans. In: 2022 International Conference on 3D Vision (3DV). pp. 414–423. IEEE (2022)

    work page 2022

  3. [3]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Aydemir, G., Akan, A.K., Güney, F.: Adapt: Efficient multi-agent trajectory predic- tion with adaptation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8295–8305 (2023) 16 Bae et al

    work page 2023

  4. [4]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Bahari, M., Saadatnejad, S., Farsangi, A.A., Moosavi-Dezfooli, S.M., Alahi, A.: Certified human trajectory prediction. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 12301–12311 (2025)

    work page 2025

  5. [5]

    Qwen2.5-VL Technical Report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    MotionMixer: MLP-based 3D Human Body Pose Forecasting

    Bouazizi, A., Holzbock, A., Kressel, U., Dietmayer, K., Belagiannis, V.: Motionmixer: Mlp-based 3d human body pose forecasting. arXiv preprint arXiv:2207.00499 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  7. [7]

    Advances in Neural Information Processing Systems36, 71862–71873 (2023)

    Cao, Y., Yihan, Z., Xu, H., Xu, D.: Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. Advances in Neural Information Processing Systems36, 71862–71873 (2023)

    work page 2023

  8. [8]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    Chen, L.H., Lu, S., Zeng, A., Zhang, H., Wang, B., Zhang, R., Zhang, L.: Mo- tionllm: Understanding human behaviors from human motions and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    work page 2025

  9. [9]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Chen, L.H., Zhang, J., Li, Y., Pang, Y., Xia, X., Liu, T.: Humanmac: Masked motion completion for human motion prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9544–9555 (2023)

    work page 2023

  10. [10]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., Chen, T.: Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26428–26438 (2024)

    work page 2024

  11. [11]

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation (2022),https://arxiv.org/ abs/2112.01527

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  12. [12]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Choi, S., Kim, J., Yun, J., Choi, J.W.: R-pred: Two-stage motion prediction via tube-query attention-based trajectory refinement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8525–8535 (2023)

    work page 2023

  13. [13]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Choudhury, R., Kitani, K.M., Jeni, L.A.: Tempo: Efficient multi-view pose estima- tion, tracking, and forecasting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14750–14760 (2023)

    work page 2023

  14. [14]

    Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes

    Delitzas, A., Parelli, M., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., Hofmann, T.: Multi-clip: Contrastive vision-language pre-training for question answering tasks in 3d scenes. arXiv preprint arXiv:2306.02329 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [15]

    IEEE transactions on pattern analysis and machine intelligence (2024)

    Delmas, G., Weinzaepfel, P., Lucas, T., Moreno-Noguer, F., Rogez, G.: Posescript: Linking 3d human poses and natural language. IEEE transactions on pattern analysis and machine intelligence (2024)

    work page 2024

  16. [16]

    In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186 (2019)

    work page 2019

  17. [17]

    Jukebox: A Generative Model for Music

    Dhariwal, P., Jun, H., Payne, C., Kim, J.W., Radford, A., Sutskever, I.: Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  18. [18]

    Project Aria: A New Tool for Egocentric Multi-Modal AI Research

    Engel, J., Somasundaram, K., Goesele, M., Sun, A., Gamino, A., Turner, A., Talattof, A., Yuan, A., Souti, B., Meredith, B., et al.: Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

    Escobar, M., Puentes, J., Forigua, C., Pont-Tuset, J., Maninis, K.K., Arbelaez, P.: Egocast: Forecasting egocentric human pose in the wild. In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 5831–5841. IEEE (2025) Ego3DLM 17

    work page 2025

  20. [20]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Fan, K., Lu, S., Dai, M., Yu, R., Xiao, L., Dou, Z., Dong, J., Ma, L., Wang, J.: Go to zero: Towards zero-shot motion generation with million-scale data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13336–13348 (2025)

    work page 2025

  21. [21]

    EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

    Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., Cao, Y.: Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  22. [22]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Feng, Y., Lin, J., Dwivedi, S.K., Sun, Y., Patel, P., Black, M.J.: Chatpose: Chatting about 3d human pose. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2093–2103 (2024)

    work page 2093

  23. [23]

    Psychonomic bulletin & review5(3), 490–495 (1998)

    Friesen, C.K., Kingstone, A.: The eyes have it! reflexive orienting is triggered by nonpredictive gaze. Psychonomic bulletin & review5(3), 490–495 (1998)

    work page 1998

  24. [24]

    Psychological bulletin133(4), 694 (2007)

    Frischen, A., Bayliss, A.P., Tipper, S.P.: Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychological bulletin133(4), 694 (2007)

    work page 2007

  25. [25]

    Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

    Fu, R., Liu, J., Chen, X., Nie, Y., Xiong, W.: Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating diverse and natural 3d human motions from text. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5152–5161 (2022)

    work page 2022

  27. [27]

    In: European Conference on Computer Vision

    Guo, C., Zuo, X., Wang, S., Cheng, L.: Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In: European Conference on Computer Vision. pp. 580–597. Springer (2022)

    work page 2022

  28. [28]

    In: Proceedings of the IEEE/CVF winter conference on applications of computer vision

    Guo, W., Du, Y., Shen, X., Lepetit, V., Alameda-Pineda, X., Moreno-Noguer, F.: Back to mlp: A simple baseline for human motion prediction. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 4809–4819 (2023)

    work page 2023

  29. [29]

    In: 2025 International Conference on 3D Vision (3DV)

    Guzov, V., Jiang, Y., Hong, F., Pons-Moll, G., Newcombe, R., Liu, C.K., Ye, Y., Ma, L.: Hmd 2: Environment-aware motion generation from single egocentric head-mounted device. In: 2025 International Conference on 3D Vision (3DV). pp. 1394–1405. IEEE (2025)

    work page 2025

  30. [30]

    The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimation

    Hatano, M., Zhu, Z., Saito, H., Damen, D.: The invisible egohand: 3d hand fore- casting through egobody pose estimation. arXiv preprint arXiv:2504.08654 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    In: Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers)

    Ho, N., Schmid, L., Yun, S.Y.: Large language models are reasoning teachers. In: Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers). pp. 14852–14882 (2023)

    work page 2023

  32. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Hong, F., Guzov, V., Kim, H.J., Ye, Y., Newcombe, R., Liu, Z., Ma, L.: Egolm: Multi- modal language model of egocentric motions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5344–5354 (2025)

    work page 2025

  33. [33]

    Advances in Neural Information Processing Systems36, 20482–20494 (2023)

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., Gan, C.: 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems36, 20482–20494 (2023)

    work page 2023

  34. [34]

    Advances in Neural Information Processing Systems37, 113991–114017 (2024)

    Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., et al.: Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems37, 113991–114017 (2024)

    work page 2024

  35. [35]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, L., Qiu, Q., Sapiro, G.: Pq-vae: Learning hierarchical discrete representations with progressive quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7550–7558 (2024) 18 Bae et al

    work page 2024

  36. [36]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Jeong, J., Lee, S., Park, D., Lee, G., Yoon, K.J.: Multi-modal knowledge distillation- based human trajectory forecasting. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 24222–24233 (2025)

    work page 2025

  37. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Jeong, J., Park, D., Yoon, K.J.: Multi-agent long-term 3d human pose forecasting via interaction-aware trajectory conditioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1617–1628 (2024)

    work page 2024

  38. [38]

    Advances in Neural Information Processing Systems36, 20067–20079 (2023)

    Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., Chen, T.: Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems36, 20067–20079 (2023)

    work page 2023

  39. [39]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Jiang, C., Cornman, A., Park, C., Sapp, B., Zhou, Y., Anguelov, D., et al.: Motion- diffuser: Controllable multi-agent motion prediction using diffusion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9644–9653 (2023)

    work page 2023

  40. [40]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kang, W., Huang, H., Shang, Y., Shah, M., Yan, Y.: Robin3d: Improving 3d large language model via robust instruction tuning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3905–3915 (2025)

    work page 2025

  41. [41]

    Land, M.F., Hayhoe, M.: In what ways do eye movements contribute to everyday activities? Vision research41(25-26), 3559–3565 (2001)

    work page 2001

  42. [42]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

    work page 2023

  43. [43]

    In: Text summarization branches out

    Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)

    work page 2004

  44. [44]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, S., Tripathi, S., Majumdar, S., Wang, X.: Joint hand motion and interaction hotspots prediction from egocentric videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3282–3292 (2022)

    work page 2022

  45. [45]

    Liu, S., Li, J., Zhao, G., Zhang, Y., Meng, X., Yu, F.R., Ji, X., Li, M.: Eventgpt: Eventstreamunderstandingwithmultimodallargelanguagemodels.In:Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29139–29149 (2025)

    work page 2025

  46. [46]

    In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp

    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi-person linear model. In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851–866 (2023)

    work page 2023

  47. [47]

    In: International Conference on Learning Representations (2017)

    Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. In: International Conference on Learning Representations (2017)

    work page 2017

  48. [48]

    In: International Conference on Learning Representations (2019)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019)

    work page 2019

  49. [49]

    In: European Conference on Computer Vision

    Ma, L., Ye, Y., Hong, F., Guzov, V., Jiang, Y., Postyeni, R., Pesqueira, L., Gamino, A., Baiyya, V., Kim, H.J., et al.: Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In: European Conference on Computer Vision. pp. 445–465. Springer (2024)

    work page 2024

  50. [50]

    Advances in Neural Information Processing Systems35, 7356–7367 (2022)

    Mao, W., Hartley, R.I., Salzmann, M., et al.: Contact-aware human motion fore- casting. Advances in Neural Information Processing Systems35, 7356–7367 (2022)

    work page 2022

  51. [51]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Mao, W., Liu, M., Salzmann, M.: Generating smooth pose sequences for diverse hu- man motion prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 13309–13318 (2021)

    work page 2021

  52. [52]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Mao, W., Liu, M., Salzmann, M., Li, H.: Learning trajectory dependencies for human motion prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9489–9497 (2019) Ego3DLM 19

    work page 2019

  53. [53]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Mao, W., Xu, C., Zhu, Q., Chen, S., Wang, Y.: Leapfrog diffusion model for stochastic trajectory prediction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5517–5526 (2023)

    work page 2023

  54. [54]

    In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)

    work page 2002

  55. [55]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Park, D., Jeong, J., Yoon, S.H., Jeong, J., Yoon, K.J.: T4p: Test-time training of trajectory prediction via masked autoencoder and actor-specific token memory. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15065–15076 (2024)

    work page 2024

  56. [56]

    In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision

    Patel, C., Nakamura, H., Kyuragi, Y., Kozuka, K., Niebles, J.C., Adeli, E.: Uniego- motion: A unified model for egocentric motion reconstruction, forecasting, and generation. In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision. pp. 10318–10329 (2025)

    work page 2025

  57. [57]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Peng, X., Mao, S., Wu, Z.: Trajectory-aware body interaction transformer for multi-person pose forecasting. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 17121–17130 (2023)

    work page 2023

  58. [58]

    In: European Conference on Computer Vision

    Pinyoanuntapong, E., Saleem, M.U., Wang, P., Lee, M., Das, S., Chen, C.: Bamm: bidirectional autoregressive motion model. In: European Conference on Computer Vision. pp. 172–190. Springer (2024)

    work page 2024

  59. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Pinyoanuntapong, E., Wang, P., Lee, M., Chen, C.: Mmm: Generative masked motion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1546–1555 (2024)

    work page 2024

  60. [60]

    In: International Joint Conference on Artificial Intelligence

    Qin, H., Hu, C., Magno, M.: Event-priori-based vision-language model for efficient visual understanding. In: International Joint Conference on Artificial Intelligence. pp. 16–30. Springer (2025)

    work page 2025

  61. [61]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Qiu, R., Gong, J., Zhang, X., Luo, S., Zhang, B., Cen, Y.: Adapting to observation length of trajectory prediction via contrastive learning. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 1645–1654 (2025)

    work page 2025

  62. [62]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    work page 2021

  63. [63]

    OpenAI blog1(8), 9 (2019)

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog1(8), 9 (2019)

    work page 2019

  64. [64]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Rahman, M.R.U., Scofano, L., De Matteis, E., Flaborea, A., Sampieri, A., Galasso, F.: Best practices for 2-body pose forecasting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3614–3624 (2023)

    work page 2023

  65. [65]

    In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). pp. 3982–3992 (2019)

    work page 2019

  66. [66]

    In: 2023 IEEE International Conference on Robotics and Automation (ICRA)

    Saadatnejad, S., Rasekh, A., Mofayezi, M., Medghalchi, Y., Rajabzadeh, S., Mordan, T., Alahi, A.: A generic diffusion-based approach for 3d human pose prediction in the wild. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 8246–8253. IEEE (2023)

    work page 2023

  67. [67]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024) 20 Bae et al

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  68. [68]

    In: European Conference on Computer Vision

    Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.: Motionclip: Exposing human motion generation to clip space. In: European Conference on Computer Vision. pp. 358–374. Springer (2022)

    work page 2022

  69. [69]

    Human Motion Diffusion Model

    Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  70. [70]

    Advances in neural information processing systems30(2017)

    Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems30(2017)

    work page 2017

  71. [71]

    Advances in neural information processing systems30(2017)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems30(2017)

    work page 2017

  72. [72]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, C., Wang, Y., Huang, Z., Chen, Z.: Simple baseline for single human motion forecasting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2260–2265 (2021)

    work page 2021

  73. [73]

    Advances in Neural Information Processing Systems37, 39784–39811 (2024)

    Wang, T., Song, Z., Lou, Z., Cui, Q., Zhang, L., Cheng, C., Wang, H., Tang, X., Li, H., Zhou, H.: Harmonizing stochasticity and determinism: Scene-responsive diverse human motion prediction. Advances in Neural Information Processing Systems37, 39784–39811 (2024)

    work page 2024

  74. [74]

    EventCLIP: Adapting CLIP for Event-based Object Recognition

    Wu, Z., Liu, X., Gilitschenski, I.: Eventclip: Adapting clip for event-based object recognition. arXiv preprint arXiv:2306.06354 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  75. [75]

    OmniControl: Control Any Joint at Any Time for Human Motion Generation

    Xie, Y., Jampani, V., Zhong, L., Sun, D., Jiang, H.: Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.08580 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  76. [76]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Xu, C., Tan, R.T., Tan, Y., Chen, S., Wang, Y.G., Wang, X., Wang, Y.: Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1410–1420 (2023)

    work page 2023

  77. [77]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Xu, Q., Mao, W., Gong, J., Xu, C., Chen, S., Xie, W., Zhang, Y., Wang, Y.: Joint-relation transformer for multi-person motion prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9816–9826 (2023)

    work page 2023

  78. [78]

    In: European Conference on Computer Vision

    Xu, R., Wang, X., Wang, T., Chen, Y., Pang, J., Lin, D.: Pointllm: Empowering large language models to understand point clouds. In: European Conference on Computer Vision. pp. 131–147. Springer (2024)

    work page 2024

  79. [79]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yue, J., Li, B., Pettré, J., Seyfried, A., Wang, H.: Human motion prediction under unexpected perturbation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1501–1511 (2024)

    work page 2024

  80. [80]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., Shan, Y.: Generating human motion from textual descriptions with discrete representations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14730–14740 (2023)

    work page 2023

Showing first 80 references.