Pith. sign in

REVIEW 3 major objections 4 minor 66 references

Poly-Autoregressive Prediction for Modeling Interactions

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Poly-autoregressive modeling — predicting an ego agent's next state from the other agents' present states — improves multi-agent forecasting on actions, trajectories, and object poses with one small transformer.

desk verdict Clever training objective, but the main comparisons leak future ground truth of other agents into PAR, so the claimed gains are unproven until that is fixed. read the letter →

arxiv 2502.08646 v1 pith:X2REIJBE submitted 2025-02-12 cs.CV

Neerja Thakkar , Tara Sadjadpour , Jathushan Rajasegaran , Shiry Ginosar , Jitendra Malik This is my paper · ORCID
classification cs.CV
keywords poly-autoregressivemodelingmulti-agentpredictionactionforecastingtrajectory6DoFposehand-objectinteractionnext-timesteptransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Poly-autoregressive (PAR) modeling forecasts one agent's future by conditioning the predictor on the ego agent's own history together with the past and current states of every other agent in the scene, instead of on the ego agent's history alone. The paper argues that single-agent autoregression is structurally insufficient for physical interactions, because the states of multiple agents change simultaneously and constrain one another. On three unrelated problems — human action forecasting, vehicle trajectory prediction, and six-degree-of-freedom object pose forecasting during hand-object interaction — a single proof-of-concept transformer trained with PAR outperforms the same model trained autoregressively in every case. The paper presents PAR not as one more task-specific model but as a general training recipe, since only tokenization and loss choice change between tasks.

What carries the argument

The central mechanism is same-agent next-timestep prediction. Given a flattened sequence of N × T state tokens, the training target for every token is shifted by N positions rather than 1, so each agent's token at time t supervises the prediction of that same agent at time t + 1; this contrasts with vanilla next-token training on a multi-agent sequence, which asks the model to predict another agent at the same timestep and, as the paper shows, confuses it. Around this shift sit two supporting components: a learned agent-ID embedding that marks which token belongs to which agent, and joint training in which losses are computed on the predicted tokens of all N agents. For the driving case study the paper adds a location positional encoding, a sin-cos embedding of each car's position relative to the ego car, which supplies location awareness on top of discrete motion tokens. The paper's ablations on AVA show that removing either the timestep shift or the agent-ID embedding lowers performance below the full PAR model, establishing both as load-bearing.

What would settle it

Rerun the three case studies in closed loop: at inference, have the model predict every agent's future jointly, feeding each agent its own previously predicted tokens and no ground-truth neighbor states, then compare average displacement error, final displacement error, and mAP with the oracle-conditioned results in Tables 1, 3, and 5. If the PAR numbers fall back to the autoregressive level, the reported gains come from accessing ground-truth neighbor futures rather than from the PAR training objective itself.

Watch

Extended reading notes

Core claim

At its core, PAR turns multi-agent interaction into a sequence-modeling problem over N × T tokens, one token per agent per timestep, and changes the prediction objective from next-token to same-agent next-timestep: the model learns to predict each agent's state at time t + 1 given all agents' states up to time t, with targets shifted by N positions and losses summed over every agent. Two ingredients carry the method: this shift-by-N objective and a learned agent-ID embedding summed into each token, which tells the transformer which physical entity each token refers to. At inference the model is run autoregressively on the ego agent, while the other agents' tokens at each future timestep are taken from ground truth. Across the three case studies the paper reports consistent gains over the single-agent autoregressive baseline — +1.9 mean average precision (mAP) overall and +3.5 mAP on two-person actions on AVA, 6.3 percent lower average displacement error and 6.4 percent lower final displacement error on nuScenes, and 8.9 percent and 41 percent relative improvements in rotation and translation error on DexYCB — and argues that the same framework, with a larger backbone and more data, is a route to general interaction-aware prediction.

Load-bearing premise

The reported gains rest on the inference-time assumption that the other agents' states are known for the whole prediction horizon; when those futures must instead be predicted from the same history, or arrive noisy, the advantage over single-agent autoregression could shrink or vanish.

Editorial extensions

If this is right

  • The same objective and architecture transfer across action labels, vehicle coordinates, and 6DoF poses with only tokenization and loss changes, so PAR is a drop-in replacement for the autoregressive training loop in interaction-prediction systems.
  • On AVA, gains concentrate on genuinely dyadic classes — listen +7.0, kiss +8.3, fight/hit +5.7 absolute mAP — implying the method's value grows with the amount of true interaction in the data.
  • Adding location information through the location positional encoding improves driving forecasts further, confirming the framework can absorb additional input modalities without architectural change.
  • Scaling the number of agents from 3 to 10 on nuScenes does not improve results, indicating that more context is not always better and that the marginal value of extra agents is task-dependent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A decisive experiment the paper does not run is closed-loop joint sampling — predicting all N agents' futures from the shared history so that no agent's ground-truth future is fed in; the drop from the oracle-conditioned numbers, if any, would isolate how much of the gain comes from the training objective versus from peeking at neighbor futures.
  • The outsized translation gain on DexYCB (41 percent) suggests PAR is most valuable when the ego agent is mechanically coupled to the other agent, making robotic manipulation and tool-use forecasting natural testbeds for a transfer claim the paper does not make.
  • The AVA ablations show the agent-ID embedding matters most when agents carry different modalities (60D action vectors versus quaternion and translation tokens), whereas it adds little in the all-cars nuScenes case; a targeted prediction is that a shared-modality variant of PAR could drop the ID embedding with no loss.
  • Because PAR is purely a training-objective change, it can be stacked on future backbones; if the mechanism captures true interaction structure, scaling the model and data should widen the PAR-over-AR gap rather than close it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Poly-Autoregressive (PAR), a next-timestep prediction framework for multi-agent interaction forecasting. PAR tokenizes the states of all N agents over time, trains a transformer with same-agent next-timestep prediction, agent-ID embeddings, and a joint loss on all agents, and at inference predicts the ego agent one timestep at a time while taking the other agents' states as conditioning context. The method is applied to three case studies with the same small transformer backbone: AVA social action forecasting, nuScenes vehicle trajectory prediction, and DexYCB hand-object pose forecasting. The central claim is that conditioning on other agents improves ego-agent forecasting, with reported gains of +1.9 mAP on AVA, 6.3% ADE/6.4% FDE on nuScenes, and 8.9%/41% improvements on DexYCB rotation/translation. However, the inference procedure described in Section 10.1 feeds the ground-truth future states of the non-ego agents at each step, which is not part of the task definition in Section 3.1 and gives PAR information unavailable to the AR baseline.

Significance. If the comparisons were clean, PAR would be a useful unifying baseline for multi-agent interaction modeling: it uses a single proof-of-concept architecture across three diverse modalities, requires only tokenization and loss changes per task, and the authors have released code. The paper is also transparent about its ablations, including agent-ID embedding and next-timestep prediction. However, the central empirical claim is currently not established because the evaluation protocol gives PAR oracle access to the future states of other agents, and the DexYCB translation comparison changes the target representation between PAR and AR. With corrected experiments that remove the oracle leakage or explicitly scope the claim to oracle-conditioned prediction, the framework could be a solid contribution; as it stands, the reported gains cannot be attributed to interaction modeling alone.

major comments (3)
  1. [Section 10.1; Section 3.1, Eq. (1)-(2)] The inference procedure contradicts the task definition. Section 10.1 states that to predict the ego at h+2, the model receives 'the ground truth of agents 1:N-1 at timestep h+1' appended to the history, and Figure 6 confirms that PAR is conditioned on other agents' ground-truth future states throughout the prediction horizon. But Eq. (1) and Eq. (2) define the conditioning set as S^{1:N-1}_{1:t-1}, i.e., only past states of the other agents. Consequently, every PAR result in Tables 1, 3, 5, 7, and 8 is obtained with information about the future of non-ego agents that the single-agent AR baseline does not receive. This makes the reported 'PAR outperforms AR' comparisons invalid as tests of interaction-conditioned forecasting. Please evaluate PAR in closed-loop mode where non-ego future states are replaced by the model's own predictions (or by a separate predictor), and/or compare against an AR model given the same oracle leakage; also report performance on the first predicted timestep, where no leakage occurs, separately from later timesteps.
  2. [Section 6.1; Table 5] The DexYCB translation comparison is not apples-to-apples. The paper states: 'For PAR we predict relative object-to-hand translations at each frame, using the current hand position as origin, while for AR, we predict absolute object translations without considering the interacting agent.' The 41% relative improvement in translation MSE may therefore be caused by the change in target representation rather than by interaction modeling, especially because the hand's future positions are provided to PAR as oracle inputs. Please add a controlled ablation with the same target for both models, e.g., AR with relative translations or PAR with absolute translations, and isolate the effect of relative encoding from the effect of adding the hand agent.
  3. [Sections 5.1 and 12.1; Table 3] The nuScenes comparison bundles at least three changes between '1-agent AR' and '3-agent PAR': the number of input agents, the location positional encoding (LPE), and oracle future non-ego tokens. Section 12.1 says the LPE is computed on the full trajectory, including future locations for agents 1 to N-1, and the token stream contains their ground-truth future acceleration tokens. Thus the 6.3% ADE and 6.4% FDE gains cannot be attributed to interaction reasoning alone. Please ablate the LPE by adding it to the 1-agent AR model, and remove the future non-ego tokens from PAR, to quantify how much of the gain is due to the added input modality and how much is due to the oracle leakage.
minor comments (4)
  1. [Abstract; Section 3.1] The abstract says PAR reasons about 'the past and current states of other interacting agents,' which matches the inference code but is inconsistent with Eq. (1)-(2) and with the problem statement in Section 3.1, where conditioning is on S^{1:N-1}_{1:t-1}. Please align the terminology and explicitly state whether current (same-timestep) future states of other agents are part of the task.
  2. [Table 3] The header 'Accleration' is a typo and should read 'Acceleration'.
  3. [Section 11.3] The sentence 'We conduct an ablation on our the agent ID embedding and next timestep prediction' is grammatically incorrect; please remove 'our' or 'the'.
  4. [Figure 9 caption] The caption text 'Add that loss is on distribution' appears incomplete or garbled; please clarify the intended statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; PAR's gains are empirical, with an oracle-future inference confound that is a validity issue, not a circular derivation.

full rationale

The paper does not derive any predicted quantity from the quantity being predicted. The PAR loss is a standard regression or classification loss on tokenized future states (e.g., MSE on 60D action tokens, cross-entropy on discrete acceleration tokens, quaternion and translation losses for DexYCB), and the evaluation compares learned models against fixed training-set baselines such as Random, NN, and Mirror. No fitted constant or post-hoc calibration is inserted to force the reported gains, and no uniqueness theorem or load-bearing self-citation is used to rule out alternatives; the cited prior work (e.g., MotionLM, humanoid locomotion as next-token prediction) is contextual rather than probative for the central claim. The one notable concern is the inference protocol in Sec. 10.1, which feeds ground-truth future tokens of agents 1:N-1 when predicting the next ego step; this is an oracle/leakage confound relative to the Sec. 3.1 task definition and may inflate PAR's advantage over AR, but it is not circularity: the ego predictions are still produced by the model and are not equal by construction to any input or to a fitted renamed target. Since no derivation step reduces to its own input, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method adds no invented entities. It rests on the practical availability of per-agent state tokens and on oracle access to other agents' futures at inference, plus a base transformer that can learn from the flattened multi-agent token stream.

free parameters (3)
  • Acceleration token bin edges = 13 bins per axis
    Hand-chosen discretization for nuScenes motion tokens; used for both AR and PAR, so it does not favor PAR, but it is a design choice.
  • LPE location quantization levels = 100 per axis
    Hand-chosen resolution for the location positional encoding in the nuScenes case study; affects the trajectory results.
  • Number of interacting agents sampled = 3 (10 in ablation)
    The paper picks the three closest vehicles; a hand-selected constant, though the 10-agent ablation shows it is not acutely sensitive.
assumptions (3)
  • domain assumption All agents are tokenizable into per-timestep state tokens
    The framework requires per-frame tracks and features for every agent; the authors note tracking failures can occur (Fig 10 example).
  • domain assumption Other agents' future states are available at inference
    Sec 10.1 feeds oracle states of agents 1:N-1 for the full future horizon; this is not flagged as a deployment limitation in the main text.
  • domain assumption A 4.4M-parameter transformer can represent the interaction dynamics
    The paper uses one small Llama-decoder for all tasks; larger backbones are left to future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Poly-Autoregressive Prediction for Modeling Interactions." pith.science (2026). https://pith.science/paper/X2REIJBE

@misc{pith2026250208646,
  author       = {Pith},
  title        = {Pith review of: Poly-Autoregressive Prediction for Modeling Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X2REIJBE}},
  note         = {Machine review of arXiv:2502.08646}
}
read the original abstract

We introduce a simple framework for predicting the behavior of an agent in multi-agent settings. In contrast to autoregressive (AR) tasks, such as language processing, our focus is on scenarios with multiple agents whose interactions are shaped by physical constraints and internal motivations. To this end, we propose Poly-Autoregressive (PAR) modeling, which forecasts an ego agent's future behavior by reasoning about the ego agent's state history and the past and current states of other interacting agents. At its core, PAR represents the behavior of all agents as a sequence of tokens, each representing an agent's state at a specific timestep. With minimal data pre-processing changes, we show that PAR can be applied to three different problems: human action forecasting in social situations, trajectory prediction for autonomous vehicles, and object pose forecasting during hand-object interaction. Using a small proof-of-concept transformer backbone, PAR outperforms AR across these three scenarios. The project website can be found at https://neerja.me/PAR/.

Figures

Figures reproduced from arXiv: 2502.08646 by the authors.

Figure 1
Figure 1. Inference for (a) autoregressive (AR) models and (b) our proposed poly-autoregressive (PAR) model. Solid indicates ground-truth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The PAR Framework. We begin by collecting a video dataset, such as AVA (top) or DexYCB (bottom). Then, using dataset labels or computer vision techniques, a trajectory of a given modality for our prediction task is extracted for each agent, such as action class labels (top) or object pose and 3D hand translation (bottom). Data is then tokenized, either through discretization or directly using continuous values, with… view at source ↗
Figure 3
Figure 3. Training with teacher forcing for (a) multi-agent next [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Action forecasting example. The distribution over ground truth actions are in white, and our predictions in red. A 6s action history (1Hz) is input, and 6s of future actions predicted. In the scene, the man and woman alternate between talking and listening. Initially, …
Figure 5
Figure 5. Figure 5: Per-class mAP for AVA 2-person actions. We see perfor￾mance improvement on almost all 2-person AVA action classes ((P) stands for “a person”). Some absolute mAP gains are particularly significant: listen to +7.0, kiss +8.3, fight/hit +5.7, talk to +4.4, hug +5.7, and h…
Figure 7
Figure 7. Figure 7: Rotation forecasting qualitative result on test set. 3D predictions are projected onto the image, isolating rotation results by showing the ground-truth translation. Incorporating the hand agent in the PAR framework (right) improves object pose prediction over object-o…
Figure 8
Figure 8. Figure 8: Translation forecasting qualitative result on test set. 3D predictions are projected onto the image, isolating translation results by showing the ground-truth rotation. Using the PAR frame￾work (right) instead of AR (left) improves object pose prediction [PITH_FULL_IM…
Figure 6
Figure 6. Figure 6: Example results from our single-agent AR model (top [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Architecture Top: PAR training with teacher forcing. Here, we see that the tokens are input to the model and projected to a token embedding dimension of size dh = 128, where embeddings such as the agent ID embedding can be summed. Then, the transformer output is sample…
Figure 10
Figure 10. Figure 10: Qualitative examples of action prediction on single￾person actions. While dance, ride and sit are not multi-person actions, our method is able to predict them more accurately in these examples (and overall by margins of +2.0, +1.0 and +2.7 mAP points respectively, see…
Figure 11
Figure 11. Figure 11: Per-class mAP on AVA single-person actions. On these actions, our PAR method is still stronger for the majority of action classes as compared to single-agent AR. For instance, we get an absolute 11.3 mAP gain on walking - people often walk in groups, so it makes sense…
Figure 12
Figure 12. Figure 12: Rotation prediction qualitative results. We show results from two videos. The projected 3D model in blue has the ground-truth translation for visualization purposes and our predicted rotation. In the top row (AR), the results depict the object of interest as the sole …
Figure 13
Figure 13. Figure 13: Translation prediction qualitative result. We show results from two videos. The projected 3D model in blue has the ground-truth rotation for visualization purposes and our predicted translation. In the top row (AR), the results depict the object of interest as the sol…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 50 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

    work page

  2. [2]

    Some informational aspects of visual percep- tion

    Fred Attneave. Some informational aspects of visual percep- tion. Psychological review, 61(3):183, 1954. 2

    work page 1954

  3. [3]

    A multimodal pre- dictive agent model for human interaction generation

    Murchana Baruah and Bonny Banerjee. A multimodal pre- dictive agent model for human interaction generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 1022–1023, 2020. 2

    work page 2020

  4. [4]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 2

    arXiv 2005

  5. [5]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 2, 6, 1

    work page 2020

  6. [6]

    Intentnet: Learning to predict intention from raw sensor data

    Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. In Con- ference on Robot Learning , pages 947–956. PMLR, 2018. 1

    work page 2018

  7. [7]

    Argoverse: 3d track- ing and forecasting with rich maps

    Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d track- ing and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 8748–8757, 2019. 1

    work page 2019

  8. [8]

    Dexycb: A benchmark for capturing hand grasping of objects

    Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021. 2, 1

    work page 2021

Show all 66 references
  1. [9]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020. 2

    2020

  2. [10]

    Interaction transformer for human reaction generation

    Baptiste Chopin, Hao Tang, Naima Otberdout, Mohamed Daoudi, and Nicu Sebe. Interaction transformer for human reaction generation. IEEE Transactions on Multimedia, pages 1–13, 2023. 2

    2023

  3. [11]

    A survey on multimodal large language models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei- Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vis...

    2024

  4. [12]

    Multimodal trajectory predictions for autonomous driving using deep convolutional networks

    Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 international conference on robotics and autom...

    2019

  5. [13]

    Poserbpf: A rao–blackwellized particle filter for 6-d object pose tracking

    Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. Poserbpf: A rao–blackwellized particle filter for 6-d object pose tracking. IEEE Transactions on Robotics, 37(5):1328–1342, 2021. 1

    2021

  6. [14]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 2, 1

    2010 arXiv

  7. [15]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichten- hofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6824–6835, 2021. 1

    2021

  8. [16]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 1

    2019

  9. [17]

    Remos: 3d motion- conditioned reaction synthesis for two-person interactions

    Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Chris- tian Theobalt, and Philipp Slusallek. Remos: 3d motion- conditioned reaction synthesis for two-person interactions. In European Conference on Computer Vision (ECCV), page 3,

  10. [18]

    Deep autoregressive networks

    Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autoregressive networks. In Inter- national Conference on Machine Learning, pages 1242–1250. PMLR, 2014. 2

    2014

  11. [19]

    Ava: A video dataset of spatio-temporally localized atomic visual actions

    Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Car- oline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE conf...

  12. [20]

    Multi-person extreme motion prediction

    Wen Guo, Xiaoyu Bie, Xavier Alameda-Pineda, and Francesc Moreno-Noguer. Multi-person extreme motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13053–13064, 2022. 2

    2022

  13. [21]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 1

    2022

  14. [22]

    A survey on trajectory-prediction methods for autonomous driving

    Yanjun Huang, Jiatong Du, Ziru Yang, Zewei Zhou, Lin Zhang, and Hong Chen. A survey on trajectory-prediction methods for autonomous driving. IEEE Transactions on In- telligent Vehicles, 7(3):652–674, 2022. 1

    2022

  15. [23]

    trajdata: A unified interface to multiple human tra- jectory datasets

    Boris Ivanovic, Guanyu Song, Igor Gilitschenski, and Marco Pavone. trajdata: A unified interface to multiple human tra- jectory datasets. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Bench- marks, New Orleans, USA, 2023. 6

    2023

  16. [24]

    Human action anticipation: A survey

    Bolin Lai, Sam Toyer, Tushar Nagarajan, Rohit Girdhar, Shengxin Zha, James M Rehg, Kris Kitani, Kristen Grau- 9 man, Ruta Desai, and Miao Liu. Human action anticipation: A survey. arXiv preprint arXiv:2410.14045, 2024. 1

    2024 arXiv

  17. [25]

    The neural autoregressive distribution estimator

    Hugo Larochelle and Iain Murray. The neural autoregressive distribution estimator. In Proceedings of the fourteenth inter- national conference on artificial intelligence and statistics , pages 29–37. JMLR Workshop and Conference Proceedings,

  18. [26]

    Deepim: Deep iterative matching for 6d pose estimation

    Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 683–698, 2018. 1

    2018

  19. [27]

    Intergen: Diffusion-based multi-human motion gener- ation under complex interactions

    Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion gener- ation under complex interactions. International Journal of Computer Vision, pages 1–21, 2024. 2

    2024

  20. [28]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 2

    2023

  21. [29]

    Long-term action forecasting using multi-headed attention- based variational recurrent neural networks

    Siyuan Brandon Loh, Debaditya Roy, and Basura Fernando. Long-term action forecasting using multi-headed attention- based variational recurrent neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2419–2427, 2022. 1

    2022

  22. [30]

    Synergy and synchrony in couple dances

    V ongani Maluleke, Lea M¨uller, Jathushan Rajasegaran, Geor- gios Pavlakos, Shiry Ginosar, Angjoo Kanazawa, and Jitendra Malik. Synergy and synchrony in couple dances. arXiv preprint arXiv:2409.04440, 2024. 2

    2024 arXiv

  23. [31]

    Wayformer: Motion forecasting via simple & efficient attention networks

    Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 2980–2987. IEEE, 2023. 1

    2023

  24. [32]

    Learning to listen: Modeling non-deterministic dyadic facial motion

    Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic facial motion. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20395–20405, 2022. 2

    2022

  25. [33]

    Can language models learn to listen? In Proceedings of the International Conference on Computer Vision (ICCV), 2023

    Evonne Ng, Sanjay Subramanian, Dan Klein, Angjoo Kanazawa, Trevor Darrell, and Shiry Ginosar. Can language models learn to listen? In Proceedings of the International Conference on Computer Vision (ICCV), 2023. 2

    2023

  26. [34]

    From audio to photoreal embodiment: Synthesizing humans in conversations

    Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. In ArXiv, 2024. 2

    2024

  27. [35]

    Scene transformer: A unified architecture for predicting multi- ple agent trajectories

    Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zheng- dong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting multi- ple agent trajectories. arXiv preprint arXi...

    2021 arXiv

  28. [36]

    Image transformer, 2018

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer, 2018. 2

    2018

  29. [37]

    Improving language understanding by genera- tive pre-training

    Alec Radford. Improving language understanding by genera- tive pre-training. 2018. 2

    2018

  30. [38]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsuper- vised multitask learners. OpenAI blog, 1(8):9, 2019. 2

    2019

  31. [39]

    Humanoid locomotion as next token prediction

    Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Ra- jasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402.19469, 2024. 2

    2024 arXiv

  32. [40]

    On the benefits of 3d pose and tracking for human action recognition

    Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Christoph Feichtenhofer, and Jitendra Malik. On the benefits of 3d pose and tracking for human action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 640–649,

  33. [41]

    An empirical study of autoregressive pre-training from videos

    Jathushan Rajasegaran, Ilija Radosavovic, Rahul Ravishankar, Yossi Gandelsman, Christoph Feichtenhofer, and Jitendra Ma- lik. An empirical study of autoregressive pre-training from videos. arXiv preprint arXiv:2501.05453, 2025. 2

    2025 arXiv

  34. [42]

    Hiera: A hierarchical vision transformer without the bells-and-whistles

    Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the bells-and-whistles. In International Conference on Machine Learning...

    2023

  35. [43]

    Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data

    Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part XVIII 16, page...

    2020

  36. [44]

    Motionlm: Multi-agent motion forecasting as language modeling

    Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579– 8...

    2023

  37. [45]

    Prediction and entropy of printed english

    Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1):50–64, 1951. 2

    1951

  38. [46]

    Duolando: Follower gpt with off-policy reinforcement learning for dance accompaniment

    Li Siyao, Tianpei Gu, Zhitao Yang, Zhengyu Lin, Ziwei Liu, Henghui Ding, Lei Yang, and Chen Change Loy. Duolando: Follower gpt with off-policy reinforcement learning for dance accompaniment. arXiv preprint arXiv:2403.18811, 2024. 2

    2024 arXiv

  39. [47]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  40. [48]

    Relational action forecasting

    Chen Sun, Abhinav Shrivastava, Carl V ondrick, Rahul Suk- thankar, Kevin Murphy, and Cordelia Schmid. Relational action forecasting. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 273–283, 2019. 1

    2019

  41. [49]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. InProceedings of the IEEE/CVF conference on computer vis...

    2020

  42. [50]

    Generative image mod- eling using spatial lstms

    Lucas Theis and Matthias Bethge. Generative image mod- eling using spatial lstms. Advances in neural information processing systems, 28, 2015. 2

    2015

  43. [51]

    Llama: Open and efficient foundation language mod- els, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste 10 Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation ...

    2023

  44. [52]

    A pose proposal and refinement network for better 6d object pose estimation

    Ameni Trabelsi, Mohamed Chaabane, Nathaniel Blanchard, and Ross Beveridge. A pose proposal and refinement network for better 6d object pose estimation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2382–2391, 2021. 1

    2021

  45. [53]

    Pixel recurrent neural networks

    A¨aron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Inter- national conference on machine learning, pages 1747–1756. PMLR, 2016. 2

    2016

  46. [54]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  47. [55]

    Gdr-net: Geometry-guided direct regression net- work for monocular 6d object pose estimation

    Gu Wang, Fabian Manhardt, Federico Tombari, and Xi- angyang Ji. Gdr-net: Geometry-guided direct regression net- work for monocular 6d object pose estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16611–16621, 2021. 1

    2021

  48. [56]

    se (3)-tracknet: Data-driven 6d pose tracking by calibrating image residuals in synthetic domains

    Bowen Wen, Chaitanya Mitash, Baozhang Ren, and Kostas E Bekris. se (3)-tracknet: Data-driven 6d pose tracking by calibrating image residuals in synthetic domains. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10367–10373. IEEE, 2020. 1

    2020

  49. [57]

    Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects

    Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ¨uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2023

  50. [58]

    Foun- dationpose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foun- dationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 17868–17879,

  51. [59]

    Reconstructing hand-held objects in 3d

    Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jitendra Malik. Reconstructing hand-held objects in 3d. arXiv preprint arXiv:2404.06507, 2024. 1

    2024

  52. [60]

    Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes

    Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199, 2017. 1

    2017 arXiv

  53. [61]

    Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting

    Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9813– 9823, 2021. 1 11 Poly-Autoregressive Prediction for M...

    2021

  54. [62]

    Related Work: Case Studies Action recognition/forecasting. Recent advancements in action recognition have significantly improved our ability to understand and classify human activities in videos, start- ing with the SlowFast network [ 16], which introduced a two-pathway approa...

  55. [63]

    Implementation details Token embeddings and loss

    Additional PAR Framework Details 10.1. Implementation details Token embeddings and loss. For discrete tokens, we use a standard learned embedding layer to convert the tokens to the hidden dimension dh of the model. To compute the loss, we use a classification loss between the ...

  56. [64]

    Additional Results on A V A Action Forecasting We see the results of our 1-agent AR and 2-agent PAR meth- ods on the A V A 1-person classes in Fig

    Additional Experimental Results 11.1. Additional Results on A V A Action Forecasting We see the results of our 1-agent AR and 2-agent PAR meth- ods on the A V A 1-person classes in Fig. 11. On the vast majority of these classes, our 2-agent PAR method is still stronger than 1-...

  57. [65]

    Additional Case Study Implementation De- tails We stabilize learning by using Exponential Moving Average (EMA) for training our experiments with a decay rate of 0.999 for action prediction and object translation/rotation estimation, and 0.9999 for car trajectory prediction. 12...

  58. [66]

    No temporal smoothing is applied, nor are any other modifications made to the results shown in the videos

    Supplementary Video Our supplementary video contains the full video for all qual- itative results shown in this paper, and additional qualitative results. No temporal smoothing is applied, nor are any other modifications made to the results shown in the videos. 6

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.