Pith. sign in

REVIEW 2 major objections 4 minor 24 references

Adding a drone-captured dense-traffic dataset to nuScenes training improves vehicle trajectory prediction and planning by up to 44.1% (ADE) and 44.3% (FDE).

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:14 UTC pith:ERYG5H5H

load-bearing objection Useful new drone dataset; headline nuScenes gain is confounded with dataset size, so the 44% number is not yet credible. the 2 major comments →

arxiv 2601.10554 v3 pith:ERYG5H5H submitted 2026-01-15 cs.CV

DeepUrban: Interaction-Aware Trajectory Prediction and Planning for Automated Driving by Aerial Imagery

Constantin Selzer , Fabian B. Flohr This is my paper
classification cs.CV
keywords DeepUrbandrone datasettrajectory predictionmotion planningautonomous drivingvulnerable road usersnuScenesScePT
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DeepUrban is a new aerial dataset of dense urban intersections, recorded by drones at about 100 meters altitude in three German locations and one US location, with 12 hours of tracked 3D objects across 14 road-user types and a 95% share of vulnerable road users. The paper's central claim is that training the ScePT prediction-and-planning model on DeepUrban scenarios together with the nuScenes benchmark reduces vehicle trajectory error on nuScenes validation by up to 44.1% in average displacement error and 44.3% in final displacement error, while improving the collision score by 49.6%. The authors interpret this as evidence that existing autonomous-driving benchmarks are deficient in dense, interaction-heavy traffic, and that aerial data can supply the missing scenarios. They also show that the dataset alone supports a state-of-the-art method, with prediction error dropping as training data increases, and that combining multiple locations improves cross-area generalization.

Core claim

The paper's central claim is that appending 505 scenarios from the Munich Tal location of DeepUrban to the 500-scenario nuScenes training set, then training the scene-consistent prediction-and-planning method ScePT, improves vehicle trajectory accuracy on nuScenes validation: ADE drops from 0.59 m to 0.33 m (44.1%) and FDE from 3.00 m to 1.67 m (44.3%), with the collision score improving by 49.6%. The same trend holds when all German or all four DeepUrban locations are added, with the largest gains from area-specific data. The dataset itself supports training: ScePT's vehicle/pedestrian ADE on Munich Tal scales from 0.18/0.34 at 25% training data to 0.13/0.25 at 100%, and qualitative example

What carries the argument

The central object is the DeepUrban V1 dataset itself: 20-second scenario windows, resampled from 12.5 Hz to 10 Hz, extracted from drone footage at about 100 m altitude over four intersections, with 3D bounding boxes and tracks produced by a proprietary autolabeling pipeline, plus OpenDRIVE, lanelet2, and VectorMap road data. Scenarios are split 80/10/10 per location, and any vehicle moving at least 5 m in the window can be treated as an ego agent, enabling multiple planning perspectives per scene. The predictive machinery is ScePT, a discrete conditional variational autoencoder that groups agents into interaction cliques via Louvain clustering on a spatiotemporal scene graph, then plans wit

Load-bearing premise

The claim that DeepUrban improves nuScenes transfers relies on the 80/10/10 scenario split keeping temporally overlapping windows from the same drone recording out of both training and validation; the paper does not describe any cleanup of near-duplicate frames across splits.

What would settle it

Compute the timestamp overlap between every training and validation scenario in DeepUrban's Munich Tal split: if any validation scenario shares more than zero frames with a training scenario, retrain ScePT on a split where such windows are removed and check whether the 44% ADE improvement on nuScenes persists. Alternatively, train ScePT on nuScenes alone and evaluate on nuScenes validation while also evaluating a nuScenes-trained model on DeepUrban validation; if the cross-dataset improvement only appears in one direction, the transfer claim is weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On nuScenes validation, training with DeepUrban's Munich Tal scenarios reduces vehicle ADE/FDE by 44.1%/44.3% and collision score by 49.6% compared with training on nuScenes alone.
  • The dataset's multi-ego-agent design means a single 20-second scene yields multiple planning examples, a property not typical of onboard-vehicle datasets.
  • Scaling experiments show that model accuracy on DeepUrban validation improves monotonically as training data increases from 25% to 100% (vehicle ADE falls from 0.18 m to 0.13 m).
  • Cross-location evaluation shows that adding area-specific data helps more for that area, but mixing all locations still improves overall generalization.
  • Qualitative results show that DeepUrban-trained models produce more conservative, pedestrian-aware trajectories, reducing predicted collision rates in dense scenes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because scenarios are extracted from continuous drone footage with up to 5 seconds of overlap, the 80/10/10 split may allow overlapping windows to fall in both training and validation; retraining with a non-overlapping split would test whether the 44% transfer gain is inflated by near-duplicate frames.
  • The transfer result is shown with ScePT alone; other interaction-aware architectures or stronger baselines may show smaller gains, and the relative improvement will likely shrink as nuScenes models improve.
  • The aerial augmentation strategy could be extended to other onboard benchmarks, but the optimal ratio of drone to onboard data, and how to balance geographic diversity, are open questions.
  • Drone data provides a top-down global view with no ego-occlusion, so models trained purely on it may need adaptation when deployed with onboard sensors; the paper does not address this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces DeepUrban V1, a drone-captured dataset of dense European urban intersections with a high share of vulnerable road users (VRUs), and integrates it into the TrajData dataloader. The authors evaluate a single state-of-the-art prediction/planning method, ScePT, on the dataset and perform cross-dataset experiments. The central claim is that adding DeepUrban scenarios to the nuScenes training set improves vehicle trajectory prediction and planning, with reported improvements up to 44.1% in ADE and 44.3% in FDE, plus a 49.6% collision-score improvement.

Significance. If the transfer result holds, DeepUrban would be a valuable complement to existing onboard-vehicle datasets, addressing the scarcity of high-density interaction scenarios in current benchmarks. The dataset's high VRU share, multi-ego-agent setup, and integration with TrajData are practical strengths, and the planned online benchmark is a useful community resource. However, the current evidence for the headline improvement is not yet convincing: the comparison is not size-matched, no variance estimates are provided, and only one model is evaluated.

major comments (2)
  1. [Section V-B.2, Table V] The central quantitative claim rests on a comparison between ScePT trained on nuScenes (N, 500 scenarios) and ScePT trained on nuScenes + DeepUrban Munich Tal (ND, ~505 additional scenarios). This roughly doubles the training data. No size-matched control (e.g., nuScenes plus an equal number of additional nuScenes scenarios or another non-DeepUrban dataset) is provided, and no repeated-seed statistics are reported. The 44.1% ADE / 44.3% FDE improvement could therefore be a data-quantity effect rather than a causal effect of DeepUrban's dense-urban content. This is a load-bearing gap for the abstract's headline claim.
  2. [Section III-B, Tables III–IV] Scenarios are 20-second windows with up to 5 seconds of overlap extracted from continuous drone recordings. The paper does not state that overlapping windows are prevented from appearing in both training and validation splits. If leakage occurs, the DeepUrban validation metrics in Tables III and IV are inflated, and any model selection based on those metrics would be compromised. The authors should either enforce and document overlap-cleared splits or quantify the overlap's effect. This issue does not directly contaminate the nuScenes transfer experiment, but it affects the dataset's own benchmark claims.
minor comments (4)
  1. [Section IV-B] In the metric definitions, the text says "x ij and y ij denote the coordinates" but only x appears in the formulas; the notation is inconsistent and should be cleaned up.
  2. [Abstract and Section V-B.2] The reported 44.1%/44.3% improvements are computed on the nuScenes validation set, but the abstract states the result without this qualification. The abstract or the results section should clearly indicate that these are validation-set numbers with no reported variance.
  3. [Figure 2] The second row shows the planned trajectory veering outside the drivable area. The explanation in the text is reasonable, but adding a label or annotation in the figure would help readers distinguish this planning artifact from a prediction error.
  4. [References, [24]] The text mentions a 'custom ScePT data split [24]' but does not describe what this split is or why it differs from the official nuScenes split. Please clarify.

Circularity Check

0 steps flagged

No significant circularity; the paper's central claim is an empirical benchmark result, not a derivation that reduces to its inputs.

full rationale

The paper's central assertion—that adding DeepUrban to nuScenes improves vehicle ADE/FDE by up to 44.1%/44.3%—is an empirical evaluation result, not a mathematical derivation. The training/evaluation protocol uses an external model (ScePT) and an external benchmark (nuScenes validation set). No equation in the paper defines the predicted ADE/FDE in terms of the input data, and no fitted parameter is renamed as a prediction. The cross-dataset comparison may be confounded by training-set size (nuScenes alone vs. nuScenes plus ~505 additional DeepUrban scenarios), but that is an attribution/correctness concern, not circularity: the measured error on nuScenes val is not equal to the input by construction. The temporal overlap between DeepUrban scenarios (up to 5 seconds in 20-second windows) is a potential data-leakage issue for DeepUrban's own validation numbers, but it does not make the headline nuScenes result circular. There are no load-bearing self-citations: references to DeepScenario and ScePT are external data/method sources, and the paper does not invoke a uniqueness theorem or prior author-derived functional form to force its conclusions. The manuscript explicitly notes that a detailed data description is deferred to a forthcoming DeepScenario publication, which is a transparency limitation but not a circular step. Overall, the derivation chain is self-contained and the reported improvement is an experimentally measured quantity rather than a tautology.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central numerical claims rest primarily on hand-chosen dataset construction parameters and unvalidated proprietary autolabeling quality; no new physical entities are introduced.

free parameters (5)
  • scenario_duration = 20 seconds
    Fixed scenario length chosen to align with nuScenes/nuPlan/Waymo; affects the set of qualifying ego agents and therefore all reported metrics.
  • ego_min_displacement = 5 meters within 20s
    Threshold for selecting ego agents; changing it changes scenario set and planning difficulty.
  • max_overlap = 5 seconds
    Maximum temporal overlap between consecutive scenarios; if overlaps cross the train/val split, metrics may be inflated.
  • resample_frequency = 10 Hz (from 12.5 Hz)
    Resampling choice to match other datasets; affects trajectory interpolation and metric values.
  • ScePT_clique_size = 4 agents
    The recommended clique size for nuScenes is used; this limits the number of interacting agents modeled, particularly in dense locations like Munich Tal.
axioms (4)
  • domain assumption DeepScenario's proprietary autolabeling pipeline produces accurate 3D tracks and boundary boxes.
    Section III-A states trajectories are extracted by DeepScenario's proprietary CV pipeline, with detailed description deferred to a future publication; no accuracy evaluation is provided.
  • domain assumption OpenDRIVE/lanelet2/VectorMap maps supplied with the raw data are accurate and complete for the recorded intersections.
    Section III-B lists the map formats; map errors would propagate into ego-agent selection, scene filtering, and evaluation.
  • domain assumption The 80/10/10 scenario split does not allow temporally overlapping scenario windows to span train and validation splits.
    Section IV states the split is per location, but the paper never states that the up-to-5s overlap between consecutive 20s scenarios is cleared across splits.
  • domain assumption ScePT's results are representative enough to support the claim about 'state-of-the-art' methods.
    Only ScePT is implemented (Section IV-A); no other SOTA prediction/planning models are compared, so the quantitative claims rest on this single baseline.

pith-pipeline@v1.3.0-alltime-deepseek · 9261 in / 11621 out tokens · 114993 ms · 2026-08-03T10:14:33.382759+00:00 · methodology

0 comments
read the original abstract

The efficacy of autonomous driving systems hinges critically on robust prediction and planning capabilities. However, current benchmarks are impeded by a notable scarcity of scenarios featuring dense traffic, which is essential for understanding and modeling complex interactions among road users. To address this gap, we collaborated with our industrial partner, DeepScenario, to develop DeepUrban-a new drone dataset designed to enhance trajectory prediction and planning benchmarks focusing on dense urban settings. DeepUrban provides a rich collection of 3D traffic objects, extracted from high-resolution images captured over urban intersections at approximately 100 meters altitude. The dataset is further enriched with comprehensive map and scene information to support advanced modeling and simulation tasks. We evaluate state-of-the-art (SOTA) prediction and planning methods, and conducted experiments on generalization capabilities. Our findings demonstrate that adding DeepUrban to nuScenes can boost the accuracy of vehicle predictions and planning, achieving improvements up to 44.1 % / 44.3% on the ADE / FDE metrics. Website: https://iv.ee.hm.edu/deepurban

Figures

Figures reproduced from arXiv: 2601.10554 by Constantin Selzer, Fabian B. Flohr.

Figure 1
Figure 1. Figure 1: DeepUrban dataset: Data pipeline showing detected [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative results on validation scenarios using ScePT: Scenario visualizations show the ego vehicle (pink), pedestrians [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Four scenarios validated on nuScenes: Trained on nuScenes(N) (left) and trained on nuScenes + DeepUrban (ND) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 7 linked inside Pith

  1. [1]

    NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles

    H. Caesar, J. Kabzn, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari, “NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles.”, in arXiv:2106.11810, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    CommonRoad: Composable Benchmarks for Motion Planning on Roads

    M. Althoff, M. Koschi, and S. Manzinger, “CommonRoad: Composable Benchmarks for Motion Planning on Roads.”, in IEEE Intell. Veh, pp. 719–726,, 2017

    2017

  3. [3]

    nuScenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom, “nuScenes: A multimodal dataset for autonomous driving.”, in Proc. IEEE CVPR, pp. 11 621–11 631, 2020

    2020

  4. [4]

    J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein, ”The inD Dataset: A drone dataset of naturalistic road user trajectories at german intersections.”, in IEEE Intell. Veh, pp. 1929–1934, 2020

    1929

  5. [5]

    INTERACTION Dataset: An INTER- national, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps

    W. Zhan, L. Sun, D. Wang et al., “INTERACTION Dataset: An INTER- national, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps.”, in arXiv:1910.03088, 2019

    Pith/arXiv arXiv 1910

  6. [6]

    Large Scale Interactive Motion Forecasting for Autonomous Driving: The W AYMO OPEN MOTION DATASET

    S. Ettinger, S. Cheng, and B. Caine et al., “Large Scale Interactive Motion Forecasting for Autonomous Driving: The W AYMO OPEN MOTION DATASET.”, in Proc. IEEE ICCV , pp. 9710–9719, 2021

    2021

  7. [7]

    Learning social etiquette: Human trajectory understanding in crowded scenes

    A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese, “Learning social etiquette: Human trajectory understanding in crowded scenes.”, in Proc. of the ECCV , pp. 549–565, 2016

    2016

  8. [8]

    Argoverse 2: Next Gen- eration Datasets for Self-Driving Perception and Forecasting

    B. Wilson, W. Qi, T. Agarwal et al., “Argoverse 2: Next Gen- eration Datasets for Self-Driving Perception and Forecasting.”, in arXiv:2301.00493, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Argoverse: 3D Tracking and Forecasting with Rich Maps

    M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays, “Argoverse: 3D Tracking and Forecasting with Rich Maps.”, in Proc. IEEE CVPR, pp. 8748–8757, 2019

    2019

  10. [10]

    You’ll never walk alone: Modeling social behavior for multi-target tracking

    S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool, “You’ll never walk alone: Modeling social behavior for multi-target tracking.”, in Proc. IEEE ICCV , pp. 261–268, 2009

    2009

  11. [11]

    Crowds by example

    A. Lerner, Y . Chrysanthou, and D. Lischinski, “Crowds by example.”, in Computer Graphics Forum, vol. 26, p. 655–664, 2007

    2007

  12. [12]

    trajdata: A Unified Interface to Multiple Human Trajectory Datasets

    B. Ivanovic, G. Song and I. Gilitschenski et al., “trajdata: A Unified Interface to Multiple Human Trajectory Datasets.”, in arXiv2307.13924, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    One Thousand and One Hours: Self-driving Motion Prediction Dataset

    J. Houston, G. Zuidhof, L. Bergamini, Y . Ye, A. Jain, S. Omari, V . Iglovikov, and P. Ondruska, “One Thousand and One Hours: Self-driving Motion Prediction Dataset.”, in CoRL, pp. 409–418, 2021

    2021

  14. [14]

    Vision meets Robotics: The KITTI Dataset

    A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets Robotics: The KITTI Dataset.”, in IJRR, pp. 1231–1237, 2013

    2013

  15. [15]

    Huang, H

    Z. Huang, H. Liu, and C. Lv, ”GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving”, in arXiv:2303.05760, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    From Prediction to Planning With Goal Conditioned Lane Graph Traversals

    M. Hallgarten, M. Stoll, and A. Zell, “From Prediction to Planning With Goal Conditioned Lane Graph Traversals.”, in arXiv:2302.07753, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Y . Chen, B. Ivanovic and M. Pavone, ”ScePT: Scene-consistent, Policy- based Trajectory Predictions for Planning”, in Proc. IEEE CVPR, pp. 17103–17112, 2022

    2022

  18. [18]

    Karkus, B

    P. Karkus, B. Ivanovic, S. Mannor, and M. Pavone, ”DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles”, in CoRL, pp. 2170-2180, 2022

    2022

  19. [19]

    Dauner, M

    D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta, ”Parting with Misconceptions about Learning-based Vehicle Motion Planning”, in CoRL, pp. 1268-1281, 2023

    2023

  20. [20]

    Y . Hu, K. Li, P. Liang, J. Qian, Z. Yang, H. Zhang, W. Shao, Z. Ding, W. Xu, and Q. Liu, ”Imitation with Spatial-Temporal Heatmap: 2nd Place Solution for NuPlan Challenge”, in Computing Research Repository”, in arXiv:2306.15700, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Dupuis, M

    M. Dupuis, M. Strobl, and H. Grezlikowski, ”OpenDRIVE 2010 and beyond - status and future of the de facto standard for the description pf road networks”, in DSC, pp. 231–242, 2010

    2010

  22. [22]

    Poggenhans, J

    F. Poggenhans, J. Pauls, J. Janosovits, S. Orf, M. Naumann, F. Kuhnt, and M. Mayr, ”lanelet2: A high-definition map framework for the future of automated driving”, in Proc. of the IEEE ITSC, pp. 1672–1679, 2018

    2018

  23. [23]

    [Online]

    ”DeepScenario Dataset: Great Munich (v2), Fabulous Sindelfingen (v2), Stunning Stuttgart (v2), Visionary San Francisco (v2).”, 2023. [Online]. Available: https://app.deepscenario.com. [Accessed: 31-Jul-2024]

    2023

  24. [24]

    Salzmann, B

    T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, ”Trajectron++: Dynamically feasible trajectory forecasting with heterogeneous data”, in Proc. of the ECCV , pp. 683–700, 2020

    2020