{"id":"3a4266da-cf82-40fc-81f3-2cf6bacd4787","arxiv_id":"2501.15842","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Trajectory prediction models trained on the larger Waymo dataset transfer poorly to Argoverse 2, and the polynomial-representation model still generalizes best.","lead":"This paper tests three trajectory prediction models across two driving datasets and finds that training on more data (Waymo) does not make models robust on Argoverse 2. The model with a polynomial representation and homogeneous augmentation still transfers best, suggesting out-of-distribution evaluation should be a standard benchmark practice.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The homogenization protocol makes the WO prediction task easier than A2 (Appendix A), so the observed 'poor generalization' when training on WO may be a task-difficulty artifact rather than a robustness finding; a complexity-matched test is needed.","rationale":"The reader's weakest assumption identifies the homogenization protocol as the key confound, and the paper's Appendix A provides direct support for that concern. The central claim depends on interpreting large OoD error increases as evidence of poor generalization and limited benefit from larger training data. If the WO task is systematically easier under the homogenized protocol, then the observed error increase on A2 is expected from task-difficulty shift, not from model-design or data-scale properties. The paper is honest about this possibility but does not resolve it, so the empirical conclusions should be treated as conditional on a controlled complexity-matched evaluation. This does not invalidate the paper's useful observation that OoD evaluation differs from ID evaluation, nor its suggestion to include OoD benchmarks; it does mean the specific finding about all models generalizing poorly from WO to A2 is not yet established. The reader's CONDITIONAL verdict is therefore appropriate, and no adjustment is needed.","tokens_in":7974,"tokens_out":3982,"duration_ms":41219,"concrete_test":"Compute the normalized deviation metric d from Appendix A Eq. (1) on the homogenized A2 validation set with t_start = 5s and compare its distribution to the WO validation set under the same protocol. If A2's distribution is substantially broader (e.g., the 90th-percentile lateral deviation is more than 30% larger), then rerun the WO-trained models on an A2 test subset filtered to match WO's d-distribution percentiles. If the OoD error increase over ID drops by more than 30% relative to the unfiltered result, the 'all models generalize poorly' conclusion is a task-difficulty artifact rather than a robustness finding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that all models trained on WO generalize poorly to A2—rests on a cross-dataset comparison whose task difficulty is not held constant. The homogenization protocol (Section 3, Table 2) extends WO history from 1.1s to 5s, and Appendix A shows this materially narrows the distribution of deviations from constant velocity over the prediction horizon, i.e., it makes the WO prediction task simpler. Models trained on this simpler WO task are then tested on A2, whose scenarios remain more demanding because A2's native history is already 5s and its focal agents are selected for harder maneuvers. The paper itself acknowledges this in Section 4.3.3, stating that the history extension 'potentially moves the challenging behaviors into the historical data.' If this task-difficulty gap explains the large OoD error increases, then the headline finding is not evidence about OoD robustness or about training-data scale; it is a predictable consequence of training on an easier distribution and testing on a harder one. The conclusion that 'more data does not automatically improve OoD robustness' is therefore not supported unless the comparison controls for prediction-task complexity. Without such a control, the paper's own Appendix A is direct evidence for the confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a cross-dataset out-of-distribution (OoD) evaluation of three trajectory prediction models (FMAE, QCNet, and the authors' EP variants) trained on Argoverse 2 (A2) and Waymo Open Motion (WO) and tested on the other dataset. Using a homogenization protocol to align histories, horizons, map elements, and focal-agent selection, the authors find that when models are trained on the smaller A2 dataset and tested on WO, the polynomial-representation model EP-Q with homogeneous augmentation generalizes best. In the reverse setting, training on the larger WO dataset and testing on A2, all models degrade substantially; EP-Q still shows the smallest degradation. The paper interprets this as evidence that more training data does not automatically improve OoD robustness and argues that OoD evaluation should complement standard in-distribution benchmarks.","tokens_in":8231,"tokens_out":4298,"duration_ms":40951,"significance":"If the finding is robust, this is a useful empirical contribution: it applies an explicit cross-dataset protocol to three modern architectures, compares representation and augmentation strategies, and introduces delta metrics for OoD robustness. The paper is transparent about its protocol and includes a concrete complexity analysis in Appendix A. It also states two alternative explanations for the main result, which is a strength. However, the central WO-trained-to-A2 conclusion is not yet causally identified because the homogenization protocol changes prediction-task difficulty between datasets, and the paper's own Appendix A documents exactly such a difference. The absence of repeated seeds and the unequal treatment of QCNet further weaken the quantitative claims. The contribution is defensible as a preliminary empirical study, but the headline conclusion needs additional controls.","major_comments":[{"comment":"The central claim that training on the larger WO dataset leads to poor OoD generalization on A2 is confounded by task complexity. The homogenization protocol extends WO history from 1.1s to 5s and caps the prediction horizon at 4.1s (Table 2). Appendix A shows via Eq. (1) and Figure 3 that the 5s-history WO prediction task has a markedly narrower distribution of deviations from constant velocity than the 1.1s-history task; Section 4.3.3 itself states that this 'potentially moves the challenging behaviors into the historical data.' Since A2 retains its native 5s history and its focal agents are selected for harder maneuvers, models trained on the easier WO task would be expected to have larger errors on A2 even with ideal OoD robustness. The conclusion that 'more data does not automatically improve OoD robustness' therefore requires a complexity-matched control, such as training also on 1.1s-history WO or reporting a constant-velocity baseline on both test sets. Without such a control, the headline finding is not yet supported.","section":"Section 3, Table 2, Appendix A"},{"comment":"All results appear to come from a single training run, with no error bars, repeated seeds, or statistical significance tests. Some in-distribution differences are very small (e.g., WO minADE1: QCNet 0.820 vs EP-Q 0.821), and the OoD deltas in Figure 2 are derived from these single runs. The ranking of EP-Q as the most robust model, and the claim that all models generalize poorly, need to be quantified against run-to-run variance; otherwise the main comparisons may reflect optimization noise rather than model properties.","section":"Section 4.1, Table 3, Figure 2"},{"comment":"QCNet was trained with truncated scene context (up to 50 agents and 80 map elements) because of GPU memory limits, while FMAE and the EP models were not truncated. This unequal treatment directly affects the cross-model comparison: Section 4.3.1 attributes QCNet's worse OoD robustness under homogeneous augmentation to this limited scenario complexity. Since the paper's conclusions compare augmentation strategies and data representations across models, QCNet's numbers are not on equal footing with the other models. The authors should either run QCNet with full context on appropriate hardware or explicitly state that all comparisons involving QCNet are upper bounds on its true performance.","section":"Section 4.1, Section 4.3.1"},{"comment":"The paper states that Appendix A supports the complexity hypothesis, but Appendix A only demonstrates that longer history narrows the distribution of constant-velocity deviations; it does not connect this distributional difference to the observed model error increases on A2. A direct test, such as comparing model errors on complexity-matched subsets of A2 and WO, would be needed to validate the proposed explanation.","section":"Section 4.3.3"}],"minor_comments":[{"comment":"The sentence describing focal-agent selection in WO says the list is unordered, so choosing the first fully observed non-ego agent corresponds to random sampling; this detail is useful, but it also means the selected WO agent is not necessarily the most challenging agent, which compounds the task-complexity issue and should be listed as an additional limitation.","section":"Section 3"},{"comment":"The label 'w/o pre-train' is ambiguous: the text should clarify whether FMAE was otherwise pre-trained (e.g., on a different dataset) and what the official FMAE configuration is, since pretraining status can materially affect both ID and OoD performance.","section":"Table 3"},{"comment":"Notation is inconsistent: the text introduces minADEK and minFDEK with K as a subscript, while Table 3 and Figure 2 use minADE1/minADE6 without subscripting. Please unify the notation across the manuscript.","section":"Section 4.1"},{"comment":"The transparent bars representing the OoD error increase are difficult to distinguish from the solid ID bars, especially in grayscale. Consider using distinct hatching or a clearer legend so the delta values are visually separable.","section":"Figure 2"},{"comment":"The phrase 'We present our experimental results to support this hypothesis in Appendix A' is misleading: Appendix A characterizes the datasets but does not present model-behavior results. Please rephrase to avoid implying causal evidence.","section":"Section 4.3.3"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is largely a follow-up to the authors' IROS paper [3], and the new WO-trained-to-A2 experiment is the main novel contribution. The self-citation pattern is heavy, but the external baselines on public datasets temper circularity concerns. The paper would be strengthened by releasing code and configuration files for reproducibility, and by reporting multiple seeds. The task-complexity confound documented in Appendix A is the key issue for the editorial decision; it is an acknowledged alternative explanation, but the conclusions are currently drawn as though the OoD effect were established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the reverse transfer direction: train on Waymo Open Motion, test on Argoverse 2. All three models degrade sharply, and the small polynomial-representation model EP-Q remains the best under OoD. That is a real, reproducible empirical pattern on public datasets, and the paper earns credit for benchmarking against external models rather than only its own.\n\nThe paper is also unusually candid. Section 4.3.3 explicitly states that extending WO history from 1.1s to 5s \"potentially moves the challenging behaviors into the historical data,\" and Appendix A gives a quantitative proxy showing that the 5s-history WO task has a narrower constant-velocity deviation distribution than the 1.1s version. That is exactly the right kind of self-diagnosis. The problem is that the abstract and title still sell the result as evidence about OoD robustness and training-data scale, when the data actually show something narrower: under this homogenization protocol, models trained on an easier prediction task do not transfer to a harder one. The \"more data does not automatically improve robustness\" takeaway is not supported without a complexity-matched control. A reader could flip the experiment and get the same pattern from task difficulty alone.\n\nOther soft spots are minor by comparison: single runs without error bars, QCNet truncated to 50 agents and 80 map elements while the other models run on full scenes, and the arbitrary choice of one fully observed non-ego focal agent in WO. These weaken the quantitative claims but do not change the basic observation.\n\nWho is this for? Researchers building cross-dataset evaluation protocols, and anyone working on trajectory prediction robustness. The paper is a useful cautionary example and the EP-Q design is worth attention. But the central claim needs to be reframed as a protocol-dependent finding, not a general law, and the authors should run a complexity-matched experiment before drawing conclusions about data scale.\n\nRecommendation: send to peer review. The work is honest, the data are real, and the main confound is identifiable and fixable. A serious referee should ask for a matched-difficulty control and error bars, but the paper deserves that round rather than a desk reject.","headline":"Honest and useful cross-dataset study, but the headline OoD conclusion is confounded by the authors' own homogenization protocol, which makes the Waymo task easier than Argoverse.","tokens_in":8759,"tokens_out":2022,"would_cite":false,"duration_ms":20735,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that training on a larger dataset does not automatically make trajectory prediction models more robust to out-of-distribution data, and that the smallest model with polynomial representation and homogeneous augmentation…","keywords":["trajectory prediction","out-of-distribution generalization","autonomous driving","inductive bias","data augmentation","polynomial representation","Argoverse 2","Waymo Open Motion"],"falsifier":"Train all models on Waymo's original 1.1-second history task instead of the homogenized 5-second history and test on Argoverse 2; if the large out-of-distribution error increase persists, the claim that more data fails to improve robustness is supported, but if it shrinks, the reported generalization failure is an artifact of task simplification.","tokens_in":7775,"feed_emoji":"🚗","tokens_out":6860,"duration_ms":60825,"temperature":0.7,"pith_summary":"This paper sets out to test whether trajectory prediction models that score well on in-distribution benchmarks also generalize to data from a different motion dataset. The authors train three models with different inductive biases on Argoverse 2 and Waymo Open Motion in both directions, using a homogenization protocol to align history length, prediction horizon, map elements, and focal-agent selection. They find that the smallest model, EP-Q, which represents trajectories as polynomials and treats focal and non-focal agents identically during training, shows the best generalization when trained on the smaller Argoverse 2 dataset and tested on the larger Waymo dataset. In the reverse direction, training on the larger Waymo dataset makes all models generalize poorly to Argoverse 2, so more training data did not deliver the expected robustness gain. The paper concludes that out-of-distribution testing should be a standard complement to in-distribution benchmarks.","feed_headline":"More driving data did not improve out-of-distribution prediction","feed_subtitle":"The smallest model with polynomial trajectories still generalized best, arguing for OoD benchmarks alongside standard splits.","key_machinery":"The argument is carried by three design mechanisms: polynomial data representation, in which trajectories are expressed as coefficients of low-degree polynomials rather than raw point sequences, imposing an inductive bias that restricts trajectory shapes; homogeneous augmentation, in which focal and non-focal agents are treated identically, each in its own coordinate frame, with equal loss weighting; and the dataset homogenization protocol that aligns history length (5 s), prediction horizon (4.1 s), map elements, and focal-agent selection across Argoverse 2 and Waymo Open Motion so cross-dataset errors can be compared. The small EP-Q model combines the first two mechanisms, and the protocol defines what counts as a fair out-of-distribution test.","core_discovery":"The central claim is that out-of-distribution robustness of trajectory predictors is governed by model design and dataset properties, not simply by training set size. With models that are competitive in in-distribution performance, the model with the strongest inductive bias, EP-Q, which uses low-degree polynomial input and output representations plus homogeneous augmentation, achieves the smallest error increase when moving from Argoverse 2 to Waymo Open Motion. In the opposite setting, all models, including EP-Q, show substantially larger error increases when trained on the larger Waymo dataset and tested on the smaller Argoverse 2 dataset, contradicting the expectation that more training data improves out-of-distribution generalization. The paper attributes this asymmetry to two dataset properties: the homogenized Waymo prediction task becomes simpler when its history is extended to 5 seconds, and Argoverse 2 has higher trajectory noise than Waymo. It argues that out-of-distribution evaluation should complement standard in-distribution benchmarks.","pith_inferences":["If the homogenization protocol is the real driver, then a fair cross-dataset test would need to match task difficulty, not just data format; the paper's own appendix shows the 5-second-history Waymo task deviates less from constant velocity, so the reported asymmetry may be partly an artifact of task simplification.","The noise-level hypothesis is testable: injecting synthetic noise into Waymo trajectories or denoising Argoverse 2 data should shift the robustness ranking if noise, rather than task complexity, is the causal factor.","A practical implication the authors leave implicit is that fleet operators should not choose a trajectory model by in-distribution benchmark rank alone, because out-of-distribution behavior can invert the ranking.","If this pattern holds more broadly, benchmark suites should include deliberately matched out-of-distribution splits with controlled difficulty and noise, rather than relying on whole-dataset swaps."],"forward_implications":["Out-of-distribution error should be reported alongside in-distribution leaderboard scores, because the model that tops an in-distribution leaderboard may not be the most robust across datasets.","Model capacity alone does not determine robustness: the smallest model with the strongest inductive bias can generalize better than much larger sequence-based models.","Increasing training data volume does not automatically improve cross-dataset robustness; the difficulty and noise level of the prediction task matter just as much.","Benchmark homogenization choices such as history length are part of the distribution, and extending Waymo's history to 5 seconds appears to make its prediction task easier, which can mask or exaggerate robustness differences.","For autonomous driving, out-of-distribution robustness should be a design goal in itself, motivating polynomial representations and homogeneous augmentation in future models."],"supporting_citations":[{"why":"Argoverse 2 dataset; provides the smaller training and test data and defines the focal-agent and map conventions used in the A2 evaluation.","marker":"[1]"},{"why":"Waymo Open Motion dataset; provides the larger training and test data and the original 1.1-second history setting that the homogenization protocol extends.","marker":"[2]"},{"why":"Authors' prior IROS paper that introduced the EP polynomial model, the three augmentation variants, and the homogenization protocol reused here.","marker":"[3]"},{"why":"Forecast-MAE, the sequence-based baseline with heterogeneous augmentation; its design and parameter count anchor the comparison.","marker":"[5]"},{"why":"QCNet, the sequence-based baseline with homogeneous augmentation; its design and scenario-complexity limits anchor the comparison.","marker":"[6]"},{"why":"Prior empirical Bayes study of noise characteristics across Argoverse 2 and Waymo Open Motion; supplies the noise-level hypothesis invoked to explain the asymmetric out-of-distribution results.","marker":"[11]"}],"fun_headline_variants":["More data doesn't improve OoD trajectory prediction","Smallest model wins on out-of-distribution robustness","Inductive bias beats big data for OoD generalization","Why more driving data fails OoD trajectory prediction","Data size less important than design for OoD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is a fair test of robustness only if the homogenization protocol makes the prediction task equally hard in both datasets, and the paper's own appendix suggests the Waymo task becomes easier when its history is extended to 5 seconds.","fun_headline_variants_meta":{"raw":{"variants":["More data doesn't improve OoD trajectory prediction","Smallest model wins on out-of-distribution robustness","Inductive bias beats big data for OoD generalization","Why more driving data fails OoD trajectory prediction","Data size less important than design for OoD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000556,"raw_usage":{"total_tokens":2619,"prompt_tokens":893,"completion_tokens":1726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1650}},"tokens_in":509,"tokens_out":1726,"duration_ms":12094,"temperature":1.0,"reasoning_tokens":1650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:53:26.293577+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train all models on Waymo's original 1.1-second history task instead of the homogenized 5-second history and test on Argoverse 2; if the large out-of-distribution error increase persists, the claim that more data fails to improve robustness is supported, but if it shrinks, the reported generalization failure is an artifact of task simplification.","supporting_citations":[{"cited_title":"Argoverse 2: Next generation datasets for self-driving perception and forecasting,","cited_arxiv_id":null,"evidence_quote":"Argoverse 2 dataset; provides the smaller training and test data and defines the focal-agent and map conventions used in the A2 evaluation."},{"cited_title":"Large scale interactive motion forecasting for autono- mous driving: The waymo open motion dataset,","cited_arxiv_id":null,"evidence_quote":"Waymo Open Motion dataset; provides the larger training and test data and the original 1.1-second history setting that the homogenization protocol extends."},{"cited_title":"Improving out-of- distribution generalization of trajectory prediction for autonomous driving via poly- nomial representations,","cited_arxiv_id":null,"evidence_quote":"Authors' prior IROS paper that introduced the EP polynomial model, the three augmentation variants, and the homogenization protocol reused here."},{"cited_title":"FMAE-noAug","cited_arxiv_id":null,"evidence_quote":"Forecast-MAE, the sequence-based baseline with heterogeneous augmentation; its design and parameter count anchor the comparison."},{"cited_title":"Forecast-mae: Self-supervised pre-training for motion forecasting with masked autoencoders,","cited_arxiv_id":null,"evidence_quote":"QCNet, the sequence-based baseline with homogeneous augmentation; its design and scenario-complexity limits anchor the comparison."},{"cited_title":"Unitraj: A unified framework for scalable vehicle trajectory prediction,","cited_arxiv_id":null,"evidence_quote":"Prior empirical Bayes study of noise characteristics across Argoverse 2 and Waymo Open Motion; supplies the noise-level hypothesis invoked to explain the asymmetric out-of-distribution results."}],"review_version":1}