REVIEW 3 major objections 5 minor 34 references
JAM: Keypoint-Guided Joint Prediction after Classification-Aware Marginal Proposal for Multi-Agent Interaction
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read JAM claims best minADE and minFDE among compared Waymo interactive predictors.
desk verdict A coherent two-stage predictor with a real reproducibility gap: the training labels for its headline classification trick are never defined, and the Table I split is labeled test while the text says validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the classification-aware mode query: a set of learnable query embeddings indexed jointly by agent and trajectory category, so the first-stage decoder is structurally required to cover every category before the joint stage sees any proposal. The second mechanism is keypoint-guided encoding, which takes predicted waypoints at 3s, 5s, and 8s from the marginal proposals, encodes them with an MLP and average pooling, and adds them back into the joint-stage mode queries. These components sit inside a transformer decoder that alternates Mode2Scene cross-attention, Mode2Mode self-attention, and Agent2Agent self-attention, ending in a Gaussian Mixture Model predictor. The training objective is a negative log-likelihood on the best Gaussian component plus a cross-entropy term for mode selection, written as Eq. (1) with a ground-truth modality category $y_{gt}$.
What would settle it
Take the paper's stated training setup with $Y_m=64$, $K_m=1$, assign a $y_{gt}$ to every training sample by nearest k-means anchor or behavioral class, and re-run training with those labels randomly permuted while keeping all other settings fixed; if the validation minADE and minFDE gains over the classification-free baseline persist unchanged, the classification-aware proposal is not the cause of the reported improvement.
Extended reading notes
Core claim
The paper's central claim is that joint prediction quality is limited by missing low-probability modes, and that forcing the proposal stage to be classification-aware fixes this. JAM splits its first-stage mode queries into $Y_m$ categories with $K_m$ queries per category, using either eight behavioral classes (stationary, straight, turns, U-turns) or 64 k-means intention anchors, so the marginal decoder must propose trajectories in every category. The second stage re-encodes these proposals, adds keypoint features extracted at 3s, 5s, and 8s, and outputs $K_j$ joint modes after agent-to-agent and mode-to-mode attention. With $Y_m=64, K_m=1$, the paper reports All(Avg) minADE of $0.8673$ and minFDE of $1.9073$ on the Waymo interactive benchmark, lower than all listed baselines including MTR++ and GameFormer, using a single model without ensembling.
Load-bearing premise
The load-bearing premise is that every training sample has a well-defined ground-truth trajectory category $y_{gt}$ for the classification-aware loss, but the paper never specifies how $y_{gt}$ is computed for the eight behavioral classes or the 64 k-means anchors; if that label assignment is arbitrary or undefined, Eq. (1) cannot be implemented as written and the main mechanism loses its stated meaning.
Editorial extensions
If this is right
- If the reported numbers reproduce, JAM is currently the best single-model approach on displacement error for two-agent interaction prediction on the Waymo interactive benchmark, while not holding the best mAP.
- Explicit mode-category coverage in a marginal proposal stage transfers to better joint refinement, so other two-stage predictors could adopt the same query-classification design.
- The ablation credits the classification-aware proposal with the larger gain and keypoint guidance with a smaller consistent gain, so both components contribute to the final result.
- Finer proposal categories (64 anchors) outperform eight behavioral classes and unclassified queries, with the most visible gains on pedestrians and cyclists.
- Because the joint decoder consumes marginal proposals, improvements in marginal mode coverage should directly feed into joint accuracy.
Reading between the lines
- A direct test of the mechanism: rerun the first-stage loss with randomly permuted category labels; if most of the minADE gain remains, the benefit comes from query diversity or anchor geometry rather than semantic trajectory types.
- The paper leaves the choice of keypoints open; replacing the fixed 3s/5s/8s waypoints with interaction-aware points such as predicted conflict or collision locations is a natural extension that could help in close-encounter scenarios.
- Since JAM's mAP trails MTR++ and BeTop despite the best displacement errors, coupling classification-aware proposals with a stronger scoring or ranking head is a promising follow-up.
- The same classification-aware proposal idea could be applied directly inside the joint stage's own mode queries, which the paper names as future work and which would test whether the taxonomy helps at the joint level as well.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes JAM, a two-stage framework for multi-agent interactive trajectory prediction. The first stage performs marginal prediction with classification-aware mode queries (eight behavioral categories or 64 k-means anchors) to preserve low-probability modes, and the second stage performs joint prediction with keypoint-guided encoding using waypoints at 3s, 5s, and 8s. Experiments on the Waymo Open Motion Dataset interactive benchmark report improved minADE and minFDE relative to several baselines, together with ablations on the proposal settings and framework components.
Significance. If the reported numbers are taken at face value, the classification-aware marginal proposal is a useful mechanism for retaining multimodal diversity in two-stage interactive prediction, and the keypoint-guided encoding is a simple and plausible addition. The availability of code, the ablation study, and the reporting of parameters and latency are strengths. However, the headline state-of-the-art claim is not fully supported by the evidence as presented: the evaluation split is ambiguous (validation versus test), the mAP is substantially below several baselines, and the ground-truth modality label used in the central loss equation is not specified. These issues place the empirical contribution on a weaker footing than the abstract suggests.
major comments (3)
- [Section IV.A.1 and Table I] The manuscript states in Section IV.A.1 that the model is 'evaluated on the validation set,' but the caption of Table I says 'TEST benchmark.' If the JAM row is computed on the validation split while the baseline rows are taken from the official test leaderboard or from test-set numbers in other papers, the comparison is cross-split and the central claim that JAM outperforms all baselines on minADE and minFDE is not established. Conversely, if all rows are validation numbers, the caption is mislabeled and comparisons to test-leaderboard results in the abstract are misleading. This issue must be resolved before the headline empirical claim can be assessed.
- [Section III.E, Eq. (1)] Equation (1) uses the ground-truth modality category ygt and the modality-specific best index kgt_y, but the paper never specifies how ygt is computed for either the eight-class behavioral taxonomy or the 64-anchor k-means clustering described in Section III.C. The proposed classification-aware marginal proposal, which is a main contribution, depends on this label existing and being computable for every training sample. Without a precise assignment rule, the loss cannot be implemented as written, and the ablation in Table III cannot be interpreted. The authors should state the exact assignment procedure, including how k-means anchors are matched to ground-truth trajectories.
- [Section IV.B and Table I] The paper claims in the abstract and in Section IV.B that JAM 'achieves state-of-the-art performance in interactive trajectory prediction,' but Table I shows All(Avg) mAP of 0.1476 for JAM versus 0.2326 for MTR++ and 0.2412 for BeTop. The improvement is limited to minADE and minFDE, and on mAP JAM is substantially worse than the strongest baselines. The state-of-the-art claim should be narrowed to the displacement-error metrics, or the mAP gap should be discussed explicitly, because mAP is a primary metric on the WOMD interactive benchmark.
minor comments (5)
- [Section III.A] The prediction horizon T is used in the definition of the output trajectories but is never defined; please specify its value and relationship to the historical horizon Th.
- [Figure 2] The legend uses Km both for 'number of mode queries for each modality classification' and in the tensor dimensions [KmYm, Ddim]; please clarify the notation so the dimension of the query tensor is unambiguous.
- [Section III.C] There is a minor typo in 'the historical encodings of the multi-agent agents,' where 'multi-agent' appears to be an unintended modifier; it should likely read 'the historical encodings of the agents.'
- [Section III.D] The sentence 'We also propose explicitly modeling key points as inputs to the model' is immediately followed by another paragraph beginning 'Additionally, we explicitly model keypoints,' which is redundant; consider merging these paragraphs.
- [Table II] The 'Joint (GameFormer)' row is described as a replication based on the official open-source implementation, but it is not stated whether this row uses the same training schedule, data split, and evaluation protocol as the other rows; please clarify so that the framework comparison is controlled.
Circularity Check
No significant circularity: the central claims are empirical comparisons against the external Waymo Open Motion Dataset benchmark, and no derivation reduces to its own inputs.
full rationale
The paper's strongest claims are empirical: JAM is reported to outperform baselines on minADE and minFDE on the WOMD interactive benchmark. These results are evaluated against ground-truth trajectories on an external dataset, not against any quantity that was fitted or defined in terms of the claimed outcome. The classification-aware marginal proposal uses 64 k-means anchors computed from the training set to define mode categories, and Eq. (1) uses an unspecified ground-truth category index ygt; however, this affects reproducibility, not circularity, because the reported metrics do not reduce to the anchor assignment or to the category labels. The keypoint-guided encoding uses fixed waypoints (3s, 5s, 8s) from the first-stage predictions as features for the second stage, which is a legitimate architectural input rather than a target being predicted. The paper contains no load-bearing self-citations, no invoked uniqueness theorem from the authors' prior work, and no renamed known result presented as a derivation. The validation/test split inconsistency in Table I is a correctness and fairness concern, not a circularity concern, because nothing in the comparison is logically forced by construction. Therefore the derivation chain is self-contained with respect to circularity and the appropriate score is 0.
Assumptions & free parameters
free parameters (4)
- marginal mode categories (Ym, Km) =
Ym=64, Km=1
- joint mode queries Kj =
6
- keypoint time steps =
3s, 5s, 8s
- number of neighboring agents =
32
assumptions (4)
- domain assumption Trajectory distributions are Gaussian mixtures with diagonal covariance.
- domain assumption k-means clustering over training-set endpoints yields a complete and balanced set of future trajectory modalities.
- ad hoc to paper A ground-truth modality category ygt exists and is computable for every training sample.
- domain assumption Validation set results on WOMD are representative of the test benchmark.
Cite this review
Pith. "Pith review of JAM: Keypoint-Guided Joint Prediction after Classification-Aware Marginal Proposal for Multi-Agent Interaction." pith.science (2026). https://pith.science/paper/BM7W2B7P
@misc{pith2026250717152,
author = {Pith},
title = {Pith review of: JAM: Keypoint-Guided Joint Prediction after Classification-Aware Marginal Proposal for Multi-Agent Interaction},
year = {2026},
howpublished = {\url{https://pith.science/paper/BM7W2B7P}},
note = {Machine review of arXiv:2507.17152}
}
read the original abstract
Predicting the future motion of road participants is a critical task in autonomous driving. In this work, we address the challenge of low-quality generation of low-probability modes in multi-agent joint prediction. To tackle this issue, we propose a two-stage multi-agent interactive prediction framework named \textit{keypoint-guided joint prediction after classification-aware marginal proposal} (JAM). The first stage is modeled as a marginal prediction process, which classifies queries by trajectory type to encourage the model to learn all categories of trajectories, providing comprehensive mode information for the joint prediction module. The second stage is modeled as a joint prediction process, which takes the scene context and the marginal proposals from the first stage as inputs to learn the final joint distribution. We explicitly introduce key waypoints to guide the joint prediction module in better capturing and leveraging the critical information from the initial predicted trajectories. We conduct extensive experiments on the real-world Waymo Open Motion Dataset interactive prediction benchmark. The results show that our approach achieves competitive performance. In particular, in the framework comparison experiments, the proposed JAM outperforms other prediction frameworks and achieves state-of-the-art performance in interactive trajectory prediction. The code is available at https://github.com/LinFunster/JAM to facilitate future research.
Figures
Reference graph
Works this paper leans on
-
[1]
X. Jia, P. Wu, L. Chen, Y . Liu, H. Li, and J. Yan, “Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 13 860–13 875, 2023
work page 2023
-
[2]
Learning lane graph representations for motion forecasting,
M. Liang, B. Yang, R. Hu, Y . Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” in Pro- ceedings of the European Conference on Computer Vision . Springer, 2020, pp. 541–556
work page 2020
-
[3]
Multimodal motion prediction with stacked transformers,
Y . Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou, “Multimodal motion prediction with stacked transformers,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 7577–7586
work page 2021
-
[4]
Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction,
B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov et al. , “Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction,” in Proceedings of the International Confer- ence on Robotics and Automation (ICRA) . IEEE, 2022, pp. 7814– 7821
work page 2022
-
[5]
Tpcn: Temporal point cloud networks for motion forecasting,
M. Ye, T. Cao, and Q. Chen, “Tpcn: Temporal point cloud networks for motion forecasting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 11 318–11 327
work page 2021
-
[6]
Vectornet: Encoding hd maps and agent dynamics from vectorized representation,
J. Gao, C. Sun, H. Zhao, Y . Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , 2020, pp. 11 525–11 533
work page 2020
-
[7]
Densetnt: End-to-end trajectory pre- diction from dense goal sets,
J. Gu, C. Sun, and H. Zhao, “Densetnt: End-to-end trajectory pre- diction from dense goal sets,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 15 303– 15 312
work page 2021
-
[8]
Implicit latent variable model for scene-consistent motion forecasting,
S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun, “Implicit latent variable model for scene-consistent motion forecasting,” in Pro- ceedings of the European Conference on Computer Vision . Springer, 2020, pp. 624–641
work page 2020
Show all 34 references
-
[9]
Scept: Scene-consistent, policy- based trajectory predictions for planning,
Y . Chen, B. Ivanovic, and M. Pavone, “Scept: Scene-consistent, policy- based trajectory predictions for planning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 17 103–17 112
2022
-
[10]
Lookout: Diverse multi-future prediction and planning for self-driving,
A. Cui, S. Casas, A. Sadat, R. Liao, and R. Urtasun, “Lookout: Diverse multi-future prediction and planning for self-driving,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 16 107–16 116
2021
-
[11]
Scene transformer: A unified multi-task model for behavior prediction and planning,
J. Ngiam, B. Caine, V . Vasudevan, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal et al. , “Scene transformer: A unified multi-task model for behavior prediction and planning,” in Proceedings of the International Conference on Learning Represen...
2021
-
[12]
M2i: From factored marginal trajectory prediction to interactive prediction,
Q. Sun, X. Huang, J. Gu, B. C. Williams, and H. Zhao, “M2i: From factored marginal trajectory prediction to interactive prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 6543–6552
2022
-
[13]
Latent variable sequential set trans- formers for joint multi-agent motion prediction,
R. Girgis, F. Golemo, F. Codevilla, M. Weiss, J. A. D’Souza, S. E. Kahou, F. Heide, and C. Pal, “Latent variable sequential set trans- formers for joint multi-agent motion prediction,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2021
2021
-
[14]
Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving,
Z. Huang, H. Liu, and C. Lv, “Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , 2023, pp. 3903–3913
2023
-
[15]
Qcnext: A next-generation framework for joint multi-agent trajectory prediction,
Z. Zhou, Z. Wen, J. Wang, Y .-H. Li, and Y .-K. Huang, “Qcnext: A next-generation framework for joint multi-agent trajectory prediction,” arXiv preprint arXiv:2306.10508 , 2023
2023 arXiv
-
[16]
Multimodal trajectory predictions for autonomous driving using deep convolutional networks,
H. Cui, V . Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric, “Multimodal trajectory predictions for autonomous driving using deep convolutional networks,” in Pro- ceedings of the International Conference on Robotics and Automation (ICR...
2019
-
[17]
Convolutional social pooling for vehicle trajectory prediction,
N. Deo and M. M. Trivedi, “Convolutional social pooling for vehicle trajectory prediction,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 1468– 1476
2018
-
[18]
Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,
Y . Chai, B. Sapp, M. Bansal, and D. Anguelov, “Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,” in Proceedings of the Conference on Robot Learning (CoRL) , 2019
2019
-
[19]
Covernet: Multimodal behavior prediction using trajectory sets,
T. Phan-Minh, E. C. Grigore, F. A. Boulton, O. Beijbom, and E. M. Wolff, “Covernet: Multimodal behavior prediction using trajectory sets,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14 074–14 083
2020
-
[20]
Lanercnn: Distributed representations for graph-centric motion forecasting,
W. Zeng, M. Liang, R. Liao, and R. Urtasun, “Lanercnn: Distributed representations for graph-centric motion forecasting,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 532–539
2021
-
[21]
Tnt: Target-driven trajectory prediction,
H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y . Shen, Y . Shen, Y . Chai, C. Schmid et al. , “Tnt: Target-driven trajectory prediction,” in Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021, pp. 895–904
2021
-
[22]
Motion transformer with global intention localization and local movement refinement,
S. Shi, L. Jiang, D. Dai, and B. Schiele, “Motion transformer with global intention localization and local movement refinement,” Ad- vances in Neural Information Processing Systems , vol. 35, pp. 6531– 6543, 2022
2022
-
[23]
Hivt: Hierarchical vector transformer for multi-agent motion prediction,
Z. Zhou, L. Ye, J. Wang, K. Wu, and K. Lu, “Hivt: Hierarchical vector transformer for multi-agent motion prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 8823–8833
2022
- [24]
-
[25]
Fjmp: Factor- ized joint multi-agent motion prediction over learned directed acyclic interaction graphs,
L. Rowe, M. Ethier, E.-H. Dykhne, and K. Czarnecki, “Fjmp: Factor- ized joint multi-agent motion prediction over learned directed acyclic interaction graphs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 13 745–13 755
2023
-
[26]
Interaction-based trajectory prediction over a hybrid traffic graph,
S. Kumar, Y . Gu, J. Hoang, G. C. Haynes, and M. Marchetti-Bowick, “Interaction-based trajectory prediction over a hybrid traffic graph,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 5530–5535
2021
-
[27]
Trajectory prediction with linguistic representations,
Y .-L. Kuo, X. Huang, A. Barbu, S. G. McGill, B. Katz, J. J. Leonard, and G. Rosman, “Trajectory prediction with linguistic representations,” in Proceedings of the International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 2868–2875
2022
-
[28]
Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention query- ing,
S. Shi, L. Jiang, D. Dai, and B. Schiele, “Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention query- ing,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 5, pp. 3955–3971, 2024
2024
-
[29]
Query-centric trajectory prediction,
Z. Zhou, J. Wang, Y .-H. Li, and Y .-K. Huang, “Query-centric trajectory prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 17 863–17 873
2023
-
[30]
R-pred: Two-stage motion prediction via tube-query attention-based trajectory refinement,
S. Choi, J. Kim, J. Yun, and J. W. Choi, “R-pred: Two-stage motion prediction via tube-query attention-based trajectory refinement,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 8525–8535
2023
-
[31]
Smartrefine: A scenario-adaptive refinement framework for efficient motion prediction,
Y . Zhou, H. Shao, L. Wang, S. L. Waslander, H. Li, and Y . Liu, “Smartrefine: A scenario-adaptive refinement framework for efficient motion prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 15 281–15 290
2024
-
[32]
Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset,
S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y . Chai, B. Sapp, C. R. Qi, Y . Zhou et al. , “Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset,” in Proceedings of the IEEE/CVF International Conference on Computer ...
2021
-
[33]
Multi-agent trajectory pre- diction with heterogeneous edge-enhanced graph attention network,
X. Mo, Z. Huang, Y . Xing, and C. Lv, “Multi-agent trajectory pre- diction with heterogeneous edge-enhanced graph attention network,” IEEE Transactions on Intelligent Transportation Systems , vol. 23, no. 7, pp. 9554–9567, 2022
2022
-
[34]
Reasoning multi-agent behavioral topology for interactive autonomous driving,
H. Liu, L. Chen, Y . Qiao, C. Lv, and H. Li, “Reasoning multi-agent behavioral topology for interactive autonomous driving,” in Advances in Neural Information Processing Systems , 2024
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.