REVIEW 4 major objections 4 minor 30 references
Where Do You Go? Pedestrian Trajectory Prediction using Scene Features
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Scene features push pedestrian path error down to 0.25 m on standard benchmarks.
desk verdict Plausible architecture, but the headline SOTA claim rests on a four-scene subset and an unspecified attention mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The cross-attention fusion module is the central mechanism: graph features from the sparse graph interaction module form the Query, while scene features from the semantic/visual extraction pipeline form the Key and Value, with a residual connection that adds the original graph features back to the attention output. This lets the model select relevant environmental cues for each pedestrian's motion while retaining its learned social interaction representation, and the refined features are passed through a temporal convolutional network to predict the future trajectory.
What would settle it
Re-run the comparison on the exact same four-scene subset for all baselines, or obtain the UNIV frames and evaluate ScenePTP on UNIV as well; if the per-scene margins over SGCN shrink or vanish when the scene sets are matched, the claimed state-of-the-art improvement is not supported.
Extended reading notes
Core claim
The paper claims that pedestrian trajectory prediction on the ETH/UCY benchmarks improves significantly when detailed scene information is added to a sparse graph interaction model. The scene representation is built by enhancing raw frames with Real-ESRGAN, running OneFormer semantic segmentation, and extracting features with ResNet-18 plus convolutional layers; these are concatenated and refined by an MLP. The graph features from SGCN serve as queries in a cross-attention layer that keys on the scene features, with a residual connection preserving the original interaction signal, and a temporal convolutional network produces the final trajectories. On the four scenes with available public frames (ETH, HOTEL, ZARA1, ZARA2), the model reports ADE/FDE of 0.252/0.372 meters, including a HOTEL ADE of 0.145 meters versus 0.31 for the SGCN baseline.
Load-bearing premise
The headline average improvement rests on comparing ScenePTP's four-scene average (ETH, HOTEL, ZARA1, ZARA2) with baselines' five-scene averages that include UNIV, so if UNIV is a harder-than-average scene, the gain may come from dropping it rather than from the method.
Editorial extensions
If this is right
- If the reported results are correct, adding semantic scene context to a sparse graph interaction model yields a large accuracy gain on the ETH/UCY subset, with HOTEL showing the most dramatic improvement.
- The ablation study indicates that semantic maps contribute consistently across all tested scenes, so future trajectory predictors should treat scene segmentation as a standard input rather than an optional extra.
- The architecture shows that a pre-trained segmentation backbone can be frozen and combined with a graph-based interaction model through cross-attention, avoiding end-to-end retraining of the segmentation stage.
- The use of image restoration before segmentation suggests that input image quality is a practical bottleneck for scene-aware trajectory prediction.
Reading between the lines
- The paper's headline average is not directly comparable to baseline averages because ScenePTP's AVG is computed over only four scenes (ETH, HOTEL, ZARA1, ZARA2) while baselines like SGCN are averaged over all five including UNIV; if UNIV is a relatively high-error scene, part of the reported margin could come from excluding it.
- The semantic segmentation backbone is trained on Cityscapes, an urban driving dataset, and its transfer to the ETH/UCY scenes may itself be a source of the gains; a segmentation model fine-tuned on those scenes might perform differently.
- The same cross-attention fusion pattern could be applied to other context modalities, such as occupancy grids, map data, or audio cues, suggesting a general recipe for conditioning graph-based motion forecasting on arbitrary environment information.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ScenePTP, a pedestrian trajectory prediction model that combines Sparse Graph Convolutional Network (SGCN) interaction features with scene features extracted via Real-ESRGAN image enhancement, OneFormer semantic segmentation, and ResNet-18 visual features. The two feature streams are fused with a cross-attention module and a temporal convolutional network predicts future positions. The authors report ADE/FDE of 0.252/0.372 on a four-scene subset of ETH/UCY (ETH, HOTEL, ZARA1, ZARA2), claiming significant improvement over state-of-the-art baselines, with the largest gain on HOTEL. A two-configuration ablation suggests that adding semantic maps improves over frame features alone.
Significance. If the reported results are reproducible and correctly attributed, the proposed integration of semantic scene features with sparse graph interactions would be a practically relevant contribution, since the claimed margins over SGCN on HOTEL and ETH are large. The paper addresses an important problem and uses standard datasets and metrics, and the high-level architecture is clearly presented. However, the significance is currently not established: the central SOTA claim depends on an incomplete evaluation (UNIV omitted), the cross-attention mechanism that is supposed to ground scene features to individual pedestrians is underspecified, no statistical validation is provided, and the ablation does not isolate the proposed scene-fusion branch from the SGCN backbone. These issues prevent the reader from assessing whether the improvement is due to the method or to evaluation and implementation choices.
major comments (4)
- [§4.1, Table 1] The standard ETH/UCY evaluation is a five-scene leave-one-out protocol, but results are reported on only four scenes because UNIV frames are said to be unavailable. I checked the AVG row in Table 1: for every listed model, including SGCN, the AVG is the mean of the same four scene columns, so the aggregate comparison is internally consistent and not an apples-to-oranges arithmetic comparison. The load-bearing problem is different: the abstract and Section 4.3 claim that the method 'significantly outperforms existing state-of-the-art approaches' on the ETH/UCY benchmark, while UNIV, a standard and typically challenging scene, is simply absent, with no result, no baseline numbers, and no code to reproduce the four-scene numbers. The SOTA claim over the full benchmark is therefore not established. The authors must either provide a UNIV result or explicitly and consistently scope every claim in the abstract, Section 4.3, and the conclusion to the four-scene subset.
- [§3.5, Eq. (1)] The cross-attention fusion is underspecified to the point that the paper's central mechanism is not a well-defined operation. The text says graph features serve as Q while scene features serve as K/V, but it never states whether H_scene is a single global vector, a flattened feature map, or a set of tokens, nor how scene feature locations are associated with each pedestrian's spatial coordinates (localized crops, homography alignment, coordinate embeddings, or a global descriptor). If H_scene is a global descriptor, then the softmax in Eq. (1) has exactly one key-value pair and the attention weight is trivially 1 for every pedestrian, so the model cannot 'prioritize relevant environmental factors that influence each pedestrian's movement' as claimed. Since the large reported HOTEL improvement (ADE 0.31 to 0.145) is attributed to exactly this mechanism, the architecture must be specified at the tensor level and ablated before the result can be interpreted.
- [§4.3, Table 1] All reported ADE/FDE numbers are single runs with no error bars, no multiple seeds, and no statistical significance test, despite the abstract and Section 4.3 using the words 'significantly' and 'significant margins.' The margins over SGCN are large on some scenes, but per-scene variance is unknown and the model's advantage is not uniform. At minimum, the authors should report mean and standard deviation over several seeds, or an equivalent confidence interval, to support the superiority claim. Without this, the reader cannot distinguish a genuine improvement from training noise or favorable initialization.
- [§4.4, Table 2] The ablation study compares 'w/o Maps' (frame features only) with 'w/ Maps' (frame features plus semantic maps), but both configurations include the full cross-attention module and the scene-feature branch. There is no ablation that removes the scene branch entirely and uses only SGCN features, so the paper never isolates the contribution of the proposed scene-feature fusion over the SGCN backbone. Given that the SGCN baseline numbers in Table 1 are taken from the original SGCN paper rather than from the authors' own reimplementation, the reported improvement cannot be attributed to the proposed module without a controlled comparison using the same training pipeline, data pre-processing, and evaluation protocol.
minor comments (4)
- [§3.6, Eq. (6)] The ADE formula sums over τ = 1 to T, but the problem definition in Section 3.1 defines future ground truth as Y_i^{t:T} = {(x_i^τ, y_i^τ) | τ = t+1, ..., T}. The summation should run over the prediction horizon (τ = t+1 to T) or use a consistently defined index set.
- [§4] The paper gives no implementation details that would allow reproduction: no batch size, learning rate schedule, number of epochs, optimizer settings (beyond 'stochastic gradient descent'), attention head count, TCN kernel size, hidden dimensions, or train/validation splitting procedure, and no code link is provided.
- [References] Reference [3] (Eigentrajectory) appears in the bibliography but is never cited in the text, and Reference [16] (Holistic LSTM) is cited but the surrounding sentence does not clearly connect to the citation.
- [§4.5–§4.6] The qualitative analysis of semantic segmentation and image restoration relies only on visual examples; adding a quantitative metric such as mIoU over the relevant scene regions would make the claimed advantage of OneFormer plus Real-ESRGAN concrete and comparable.
Circularity Check
No circularity identified: the trajectory predictions are evaluated on held-out scenes and no load-bearing step reduces to its own inputs.
full rationale
The paper's derivation chain is self-contained and empirically anchored. Scene features are extracted by fixed external pretrained models (Real-ESRGAN [23], OneFormer [9], ResNet-18 [7]) and by trainable layers optimized on the training folds; interaction features come from the external SGCN [20]. The cross-attention fusion (Eq. 1-3) is a standard attention operation with Q equal to graph features and K/V equal to scene features, and the final predictor is a TCN (Eq. 4). The loss (Eq. 5-7) is the standard ADE/FDE against ground-truth future positions, so the model is not fit to test outputs. The leave-one-out protocol trains on four scenes and tests on the remaining one, and Table 1's AVG column is computed over the same four scenes for every listed model, so the comparison is internally consistent. The exclusion of UNIV is explicitly disclosed ('Due to the absence of publicly available frame data for the UNIV scene... we exclude it') and is a completeness/protocol limitation, not a circularity. There are no self-citations, no fitted parameter renamed as a prediction, no imported uniqueness theorem, and no known result repackaged under new coordinates. The methodological concerns about the missing UNIV result and underspecified spatial grounding of H_scene belong to correctness and validity, not circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption OneFormer semantic segmentation trained on Cityscapes transfers to the low-resolution ETH/UCY scenes without fine-tuning.
- domain assumption Frame images from the ETH/UCY datasets can be aligned with trajectory timestamps at the 8/12-step protocol, and image features describe the same physical scene as the coordinates.
- domain assumption Averaging ADE/FDE over the four available scenes is comparable to the five-scene average reported for baselines.
- domain assumption Global scene features can be associated with each pedestrian via cross-attention without explicit spatial grounding (queries = graph features, keys/values = scene features).
Cite this review
Pith. "Pith review of Where Do You Go? Pedestrian Trajectory Prediction using Scene Features." pith.science (2026). https://pith.science/paper/NDZQKH5X
@misc{pith2026250113848,
author = {Pith},
title = {Pith review of: Where Do You Go? Pedestrian Trajectory Prediction using Scene Features},
year = {2026},
howpublished = {\url{https://pith.science/paper/NDZQKH5X}},
note = {Machine review of arXiv:2501.13848}
}
read the original abstract
Accurate prediction of pedestrian trajectories is crucial for enhancing the safety of autonomous vehicles and reducing traffic fatalities involving pedestrians. While numerous studies have focused on modeling interactions among pedestrians to forecast their movements, the influence of environmental factors and scene-object placements has been comparatively underexplored. In this paper, we present a novel trajectory prediction model that integrates both pedestrian interactions and environmental context to improve prediction accuracy. Our approach captures spatial and temporal interactions among pedestrians within a sparse graph framework. To account for pedestrian-scene interactions, we employ advanced image enhancement and semantic segmentation techniques to extract detailed scene features. These scene and interaction features are then fused through a cross-attention mechanism, enabling the model to prioritize relevant environmental factors that influence pedestrian movements. Finally, a temporal convolutional network processes the fused features to predict future pedestrian trajectories. Experimental results demonstrate that our method significantly outperforms existing state-of-the-art approaches, achieving ADE and FDE values of 0.252 and 0.372 meters, respectively, underscoring the importance of incorporating both social interactions and environmental context in pedestrian trajectory prediction.
Figures
Reference graph
Works this paper leans on
-
[1]
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. 2016. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition. 961–971
work page 2016
-
[2]
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2017. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE trans- actions on pattern analysis and machine intelligence 39, 12 (2017), 2481–2495
work page 2017
-
[3]
Inhwan Bae, Jean Oh, and Hae-Gon Jeon. 2023. Eigentrajectory: Low-rank descriptors for multi-modal trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 10017–10029
work page 2023
-
[4]
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus En- zweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3213–3223
2016
-
[5]
Mahsa Golchoubian, Moojan Ghafurian, Kerstin Dautenhahn, and Nasser Lash- garian Azad. 2023. Pedestrian trajectory prediction in pedestrian-vehicle mixed environments: A systematic review. IEEE Transactions on Intelligent Transporta- tion Systems (2023)
work page 2023
-
[6]
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi
-
[7]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778
2016
-
[8]
Dirk Helbing and Peter Molnar. 1995. Social force model for pedestrian dynamics. Physical review E 51, 5 (1995), 4282
work page 1995
Show all 30 references
-
[9]
Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. 2023. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2989–2998
2023
-
[10]
Hezheng Lin, Xing Cheng, Xiangyu Wu, and Dong Shen. 2022. Cat: Cross atten- tion in vision transformer. In 2022 IEEE international conference on multimedia and expo (ICME). IEEE, 1–6
2022
-
[11]
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel
-
[12]
Taylor Mordan, Matthieu Cord, Patrick Pérez, and Alexandre Alahi. 2021. De- tecting 32 pedestrian attributes for autonomous vehicles. IEEE transactions on intelligent transportation systems 23, 8 (2021), 11823–11835
2021
-
[13]
Brian Paden, Michal Čáp, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli
-
[14]
Darsh Parekh, Nishi Poddar, Aakash Rajpurkar, Manisha Chahal, Neeraj Kumar, Gyanendra Prasad Joshi, and Woong Cho. 2022. A review on autonomous vehicles: Progress, methods and challenges. Electronics 11, 14 (2022), 2162
2022
-
[15]
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. 2009. You’ll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th international conference on computer vision . IEEE, 261–268
2009
-
[16]
Ruijie Quan, Linchao Zhu, Yu Wu, and Yi Yang. 2021. Holistic LSTM for pedestrian trajectory prediction. IEEE transactions on image processing 30 (2021), 3229–3239
2021
-
[17]
Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K Tsotsos. 2019. Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6262–6271
2019
-
[18]
Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese
-
[19]
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. 2019. Sophie: An attentive gan for predict- ing paths compliant to social and physical constraints. In Proceedings of the IEEE/CVF conference on computer vision and pattern ...
2019
-
[20]
Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua. 2021. SGCN: Sparse graph convolution network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8994–9003
2021
-
[21]
Arsal Syed and Brendan Tran Morris. 2019. SSeg-LSTM: Semantic scene segmen- tation for trajectory prediction. In 2019 IEEE Intelligent Vehicles Symposium (IV) . IEEE, 2504–2509
2019
-
[22]
In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14
Learning social etiquette: Human trajectory understanding in crowded scenes. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 . Springer, 549–565
2016
-
[23]
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. 2021. Real-esrgan: Train- ing real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision . 1905–1914
2021
-
[24]
Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. 2019. A review of recurrent neural networks: LSTM cells and network architectures. Neural computation 31, 7 (2019), 1235–1270
2019
-
[25]
Simone Zamboni, Zekarias Tilahun Kefato, Sarunas Girdzijauskas, Christoffer Norén, and Laura Dal Col. 2022. Pedestrian trajectory prediction with convolu- tional neural networks. Pattern Recognition 121 (2022), 108252
2022
-
[26]
Arsal Syed and Brendan Tran Morris. 2023. Semantic scene upgrades for trajectory prediction. Machine vision and applications 34, 2 (2023), 23
2023
-
[30]
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. 2017. Pyramid scene parsing network. InProceedings of the IEEE conference on computer vision and pattern recognition . 2881–2890
2017
-
[2016]
IEEE Transactions on intelligent vehicles 1, 1 (2016), 33–55
A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles 1, 1 (2016), 33–55
2016
-
[2018]
In Proceedings of the IEEE conference on computer vision and pattern recognition
Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2255–2264
-
[2020]
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 14424–14432
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.