{"id":"5e9c023c-ecc8-4102-8a6b-d53753cf6cd9","arxiv_id":"2507.05098","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Dataset design study shows extra map and agent features do not improve the QCNet trajectory predictor, while multi-country training data improves cross-region generalization.","lead":"A Bosch team built a new self-driving trajectory prediction dataset from drives in Germany and the US and tested what dataset choices matter. They find that adding extra map and agent features does not improve a top prediction model, while training on data from multiple countries helps. A smart generalist would read this for concrete guidance on where to spend effort when designing autonomous-driving data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The feature-null result is underdetermined: the paper cannot distinguish 'QCNet already infers these features' from 'QCNet ignores the added input channels,' so a positive-control feature is needed before concluding that extra features are unnecessary.","rationale":"The reader's weakest assumption was that the added features are actually usable by QCNet. My analysis agrees that this is the most load-bearing point: the central abstract claim is a universal statement about modern architectures not needing extensive feature sets, but the evidence is a null result from one model whose feature encoder is neither described nor released. The paper's own admitted ambiguity between 'network infers the features' and 'network cannot use the features' means the conclusion overreaches. I would keep the CONDITIONAL verdict rather than moving to REJECT, because the concern is addressable in principle with a positive-control experiment and the remaining transfer and geographic-diversity results are still useful and internally consistent. The conditionality should explicitly require the positive-control feature test, multiple seeds for Table II, and ideally a release or detailed specification of the feature encoding modifications. Since the reader already assigned CONDITIONAL and my concern reinforces the same condition rather than shifting the verdict, I selected UNCHANGED.","tokens_in":9529,"tokens_out":5368,"duration_ms":68265,"concrete_test":"Run a positive control through the exact same feature pipeline: add one scalar agent feature to the training split that is strongly and causally informative, for example an indicator of whether the agent will be stationary within the next three seconds, or the ground-truth future endpoint offset. Retrain QCNet with identical hyperparameters and evaluate on the validation split. If this oracle feature fails to improve over Baseline, the added-feature pathway is not transmitting information to the predictor, so the Table II null is uninformative about feature redundancy. If it does improve, the null for the real proposed features becomes interpretable. As a secondary check, report per-channel gradient norms or attention weights for the added feature columns to confirm they are non-zero in the forward pass.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing issue is that the Section IV-A null result cannot distinguish 'QCNet already infers speed limits, stop lines, and similar attributes from geometry' from 'the customized feature encoder never exposes these inputs to the network in a usable way.' The paper itself states this ambiguity in Section IV-A: 'Either, the network is already powerful enough to infer the additional features from the limited baseline features or it is not capable of leveraging the additional features.' Yet the conclusion in Section V draws the first reading as established fact, claiming that supplementary features 'did not measurably improve prediction accuracy' and that this 'validates the sufficiency of existing public datasets.' The evaluation never specifies the 'small modifications' used to feed the extra map and agent attributes into QCNet, and the proprietary dataset and code are not released, so the feature-encoding pathway is unverifiable. If the added channels are dropped, mis-normalized, or not connected to the relevant QCNet encoder modules, the null result would be an artifact of the implementation rather than evidence about feature sufficiency. Additionally, Table II reports only point estimates without seed-level variance, so differences such as minFDE1 of 2.39 versus 2.37 cannot be ruled out as training noise; the claim of 'no measurable improvement' lacks a statistical grounding even under the assumption that the features were correctly encoded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the L4 Motion Forecasting dataset, recorded with a sensor-equipped test vehicle in Stuttgart (Germany) and Sunnyvale (USA), and structured to be compatible with Argoverse 2. Using QCNet as the evaluation model, the authors report three main findings: (i) adding enriched map and agent features (e.g., speed limits, stop/yield lines, Frenet attributes, geometric details) yields no measurable improvement over baseline features; (ii) cross-dataset transfer between L4 and Argoverse 2 shows asymmetric degradation, with pre-training on external data and fine-tuning on the target set mitigating the domain shift; and (iii) training on the combined multi-country dataset outperforms country-specific training. The paper concludes that dataset designers should prioritize geographic and scenario diversity over feature quantity.","tokens_in":9796,"tokens_out":5398,"duration_ms":62371,"significance":"If the headline results were rigorously established, they would be practically important for autonomous-driving dataset design: they would imply that the feature sets provided by public benchmarks are sufficient for state-of-the-art trajectory predictors and that multi-country diversity matters more than annotation richness. The paper's strengths include the creation of a new multi-country dataset with an Argoverse-2-compatible structure, the explicit manipulation of feature subsets, and the cross-dataset and cross-country transfer experiments. However, the current manuscript does not supply the statistical evidence needed to support the central negative feature result, the feature-encoding pathway is not audited, and the geographic-diversity conclusion is confounded with data volume. The dataset and code are not released, further limiting verification. The paper acknowledges the core ambiguity in Section IV-A but does not resolve it before drawing conclusions in Section V.","major_comments":[{"comment":"The 'no measurable improvement' claim is not statistically grounded. All metric differences across feature sets are in the 0.01-0.02 range (e.g., minFDE1 2.39 vs. 2.37; minADE1 0.99 vs. 1.01), but no training seeds, variance estimates, confidence intervals, or significance tests are reported. Without these, the observed differences could easily be training noise, and the claim should either be supported by multiple-seed statistics or rephrased as 'no improvement was observed in this single run.'","section":"Section IV-A, Table II"},{"comment":"The feature-encoding pathway for the added L4 features is never specified. The paper only says that 'small modifications are required' to feed the extra features into QCNet, and neither the code nor the dataset is released. The paper itself acknowledges the ambiguity in Section IV-A ('Either, the network is already powerful enough to infer the additional features from the limited baseline features or it is not capable of leveraging the additional features'), yet Section V concludes the first reading as established fact. A positive control is needed—for example, a feature whose effect can be independently verified—to show the added channels are actually consumed by the model; without it, the null result may be an artifact of input channels that the customized encoder never exposes to the network in a usable way.","section":"Section IV-A and Section V"},{"comment":"The conclusion that geographic diversity improves generalization is confounded with dataset size. The full L4 training set is larger than either country subset, and the Stuttgart subset is larger than the Sunnyvale subset, so the superior performance of the full- and Stuttgart-trained models could reflect more training data rather than diversity. The statement that 'a more diverse dataset spanning multiple countries' is the cause requires matched-size experiments, for example comparing a Stuttgart-only model trained on the same number of scenarios as the full set or a downsampled multi-country set.","section":"Section IV-C, Table IV"},{"comment":"The paper generalizes from a single architecture, QCNet, to 'modern architectures' and to dataset-design recommendations. Since the additional-feature null result may depend on how QCNet's encoders handle extra inputs, this generalization is not supported by the experiments. At minimum, the conclusions should be restricted to QCNet, or a second architecture should be evaluated to test whether the null result is model-specific.","section":"Section V and Abstract"}],"minor_comments":[{"comment":"The 150 ms temporal-discontinuity threshold and the 5 m stationary-trajectory cutoff are presented without justification or sensitivity analysis; since these thresholds shape the dataset statistics, a brief discussion would improve the manuscript.","section":"Section III-C2"},{"comment":"The quality-control description does not state how tracking errors or inconsistent agent IDs were identified; some details on the filtering procedure would improve reproducibility.","section":"Section III-C1"},{"comment":"Reference [17] appears to have a garbled author list ('Kong, and D. Global') and an incomplete title; please correct it.","section":"Reference list"},{"comment":"The claim that the L4 dataset 'poses greater inherent challenges' is inferred from lower metric values; this should be framed as a hypothesis, since evaluation difficulty can be affected by dataset composition, label quality, and scenario mix.","section":"Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"This is an industry dataset paper whose dataset is proprietary and not released. That is a serious limitation for a journal publication, and the editors may wish to consider whether the contribution can be adequately evaluated without data access. The statistical and positive-control issues are fixable but require additional experiments, so I recommend major revision rather than rejection: the underlying question is relevant, and the cross-country transfer experiments have potential value if the confounding factors are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: a solid dataset paper with a genuinely new resource and mostly honest interpretation, but the headline claim—that extra features don't matter—is underdetermined by the evidence as presented. It deserves a referee, but needs work.\n\nThe new L4 Motion Forecasting dataset is the real contribution. 90k scenarios from Germany and the US, 274 hours, with Frenet attributes, speed limits, stop lines, and other extras. Structuring it in Argoverse 2 format means QCNet runs on it with minimal modification. That alone is useful to the motion forecasting community, even though the data isn't released.\n\nThe paper also does something useful: a clean ablation over feature subsets, cross-dataset transfer between AV2 and L4, and a Germany/US split. The cross-dataset and country-transfer results are worth reading, and the observation that more diverse training data helps even when one country dominates is a concrete, actionable finding for dataset builders.\n\nThe soft spots are real. First, the feature-null result: Table II differences are 0.01–0.02 in the metrics, with no error bars, multiple seeds, or significance tests. With one run per configuration, 'no measurable improvement' isn't statistically grounded. Second, the paper's own Section IV-A concedes the ambiguity—either QCNet already infers these features, or it doesn't use the new channels—but Section V asserts the first reading as established. That's a leap. The 'small modifications' to feed extra features are never specified, and the dataset and code aren't public, so the encoding pathway is unverifiable. A positive control would settle this. Third, the geographic diversity conclusion confounds country with data volume and composition; Stuttgart dominates, so the Sunnyvale-only model's poor performance could just be size, not culture.\n\nNone of these are fatal. The dataset analysis and transfer numbers stand on their own, and the qualitative roundabout examples are suggestive. The paper needs multi-seed runs, error bars, a clear description of the feature-encoding modifications, and a more careful framing of the null result.\n\nRecommendation: send to peer review. It deserves referee time, with major revisions expected. I'd bring it to reading group only if we're discussing dataset design, and I'd cite it cautiously, mostly for the transfer findings.","headline":"A useful new dataset and some genuinely informative transfer experiments, but the headline feature-sufficiency claim is underdetermined and needs more statistical and architectural support.","tokens_in":10320,"tokens_out":2788,"would_cite":true,"duration_ms":28530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding richer lane, stop-line, speed-limit, Frenet, and agent-geometry features to a top trajectory prediction model produces no measurable gain, and that dataset builders should invest in geographic and scenario…","keywords":["multi-agent trajectory prediction","dataset design","feature selection","cross-dataset transfer","geographic diversity","autonomous driving","QCNet","Argoverse 2"],"falsifier":"Train the same QCNet configuration on the L4 splits with a feature-blind control (e.g., constant speed limits and constant stop-line distances) and compare with the full-feature model: identical metrics would show the model ignored the extra inputs, while a difference would show the features are usable and the null result needs re-examination. Stronger still, rerun the same six feature configurations on a second architecture with an explicit lane-graph encoder; any repeatable improvement beyond run-to-run noise on the Argoverse 2 metrics would overturn the paper's general conclusion.","tokens_in":9343,"feed_emoji":"🚗","tokens_out":9413,"duration_ms":94536,"temperature":0.7,"pith_summary":"The paper tries to establish that dataset design decisions, not just model architecture, determine how well multi-agent trajectory prediction performs and transfers. It introduces the L4 Motion Forecasting dataset, built from two years of German and US driving recordings and formatted to match the Argoverse 2 benchmark, and evaluates the QCNet model on it. The central finding is that adding supplementary map and agent features—extra lane segments, speed limits, stop and yield lines, Frenet attributes, distances to road objects, and finer geometry—produces no measurable improvement over the baseline feature set on any reported metric. The authors conclude that modern architectures do not need extensive feature sets and that dataset builders should put their effort into geographic and scenario diversity. Supporting experiments show that pretraining on a larger US-centric dataset and fine-tuning on L4 preserves most target performance, and that a multi-country training set helps each individual region.","feed_headline":"Extra lane and agent features don't improve trajectory prediction","feed_subtitle":"A Germany-US driving dataset shows the public feature set is enough; diversity beats extra annotations.","key_machinery":"The load-bearing mechanism is the controlled feature-ablation setup. Because the L4 dataset is constructed in the Argoverse 2 schema, the same public QCNet implementation can be trained on baseline features without modification and on expanded features with only small changes, isolating feature content while architecture, splits, and evaluation are fixed. The evaluation itself uses the Argoverse 2 benchmark metrics—b-minFDE, minADE6, minFDE6, MR6, minADE1, minFDE1, MR1—averaged over all scored agents, so the feature comparison is measured on the same scale as public leaderboards.","core_discovery":"On the L4 dataset, the paper compares six feature configurations—Baseline, Enhanced Lanes, Enhanced Stop Lines, Enhanced Agents, Full Integration, and Focused Integration—all trained with the same QCNet model and evaluated with the Argoverse 2 competition metrics. The numbers in Table II move only by a few hundredths of a meter (b-minFDE stays at 1.29–1.30 m and minADE6 at 0.47 m across all variants), which the authors read as no measurable benefit from the added information. Cross-dataset experiments then show that a model trained only on Argoverse 2 loses more when tested on L4 than an L4-trained model loses when tested on Argoverse 2, with L4's wider speed distribution and roundabouts cited as the reason. Splitting L4 by country, a Stuttgart-trained model beats a Sunnyvale-trained model even on the Sunnyvale test set, while the model trained on both countries is best on both. The paper concludes that richer features are unnecessary for current architectures and that geographic diversity is the more valuable dataset property.","pith_inferences":["The paper does not test whether the added features would help simpler architectures; its null result is specific to QCNet, so a useful extension is repeating the same feature ablation with a non-transformer baseline or a model with an explicit lane-graph encoder.","The averaged competition metrics may mask safety-critical differences; counting violations of stop/yield rules at roundabout entries or unsignalized intersections would be a sharper test of whether stop-line information matters in edge cases.","If the finding generalizes, the expensive parts of HD-map annotation (speed limits, stop lines, lane-subtype labels) would be low-value for trajectory prediction, and dataset cost could be cut by recording more diverse routes instead of annotating the same routes more finely.","One can make the claim falsifiable in an inexpensive way by adding a dropout or noise injection on the extra features: if the model's predictions are unchanged, the features are not being used at all."],"forward_implications":["The limited feature sets of public datasets suffice for state-of-the-art trajectory predictors, so dataset creators can redirect annotation effort toward more scenarios rather than more attributes per scenario.","Pretraining on a large, geographically different dataset and fine-tuning on the target dataset recovers most of the target-domain accuracy while retaining knowledge of the source domain.","Geographic diversity in training data improves performance on every region tested, even when one region dominates the data, so adding a smaller dataset from another country is a worthwhile investment.","Domain shift between datasets is asymmetric: a model trained on the broader, higher-speed L4 data transfers to Argoverse 2 better than an Argoverse 2-trained model transfers to L4.","Roundabouts and high-speed rural driving, underrepresented in US-centric benchmarks, are the kinds of scenarios that most expose cross-dataset transfer failures."],"supporting_citations":[{"why":"Supplies the QCNet query-centric transformer architecture used for all feature, transfer, and location experiments.","marker":"[6]"},{"why":"Defines the Argoverse 2 dataset and schema that the L4 dataset mirrors, and provides the comparison benchmark.","marker":"[1]"},{"why":"Defines the competition metrics (b-minFDE, minADE6, minFDE6, MR6, minADE1, minFDE1, MR1) used to compare all models.","marker":"[7]"},{"why":"Provides the public QCNet implementation the paper trains, so the feature ablations use one fixed codebase.","marker":"[19]"},{"why":"Introduced HD-map-based forecasting and motivates why map features are expected to matter, which the paper tests.","marker":"[4]"},{"why":"Source of the Argoverse 2 distribution figures (maneuvers, actors per scenario) that the paper plots against L4.","marker":"[20]"}],"fun_headline_variants":["Extra map features don't boost trajectory prediction","Dataset diversity beats feature richness for trajectory models","More features, same accuracy: why dataset design matters more","Cross-dataset tests show geographic variety trumps extra annotations","Trajectory prediction: feature set is enough, diversity is key"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that QCNet actually consumes the added features—the paper reports only that small modifications are needed, without validating the feature encoder—and that one model's non-reaction means modern architectures in general do not need richer features.","fun_headline_variants_meta":{"raw":{"variants":["Extra map features don't boost trajectory prediction","Dataset diversity beats feature richness for trajectory models","More features, same accuracy: why dataset design matters more","Cross-dataset tests show geographic variety trumps extra annotations","Trajectory prediction: feature set is enough, diversity is key"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2264,"prompt_tokens":929,"completion_tokens":1335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1258}},"tokens_in":545,"tokens_out":1335,"duration_ms":11135,"temperature":1.0,"reasoning_tokens":1258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:32:12.418743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same QCNet configuration on the L4 splits with a feature-blind control (e.g., constant speed limits and constant stop-line distances) and compare with the full-feature model: identical metrics would show the model ignored the extra inputs, while a difference would show the features are usable and the null result needs re-examination. Stronger still, rerun the same six feature configurations on a second architecture with an explicit lane-graph encoder; any repeatable improvement beyond run-to-run noise on the Argoverse 2 metrics would overturn the paper's general conclusion.","supporting_citations":[{"cited_title":"Query-Centric Trajectory Prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the QCNet query-centric transformer architecture used for all feature, transfer, and location experiments."},{"cited_title":"Argoverse 2: Next generation datasets for self-driving perception and forecasting,","cited_arxiv_id":null,"evidence_quote":"Defines the Argoverse 2 dataset and schema that the L4 dataset mirrors, and provides the comparison benchmark."},{"cited_title":"Argoverse 2: Motion Forecasting Competition,","cited_arxiv_id":null,"evidence_quote":"Defines the competition metrics (b-minFDE, minADE6, minFDE6, MR6, minADE1, minFDE1, MR1) used to compare all models."},{"cited_title":"QCNet: An Elegant, Performant, And Scalable Framework For Marginal/Joint Multi-Agent Trajectory Prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the public QCNet implementation the paper trains, so the feature ablations use one fixed codebase."},{"cited_title":"Argoverse: 3D Tracking and Forecasting With Rich Maps,","cited_arxiv_id":null,"evidence_quote":"Introduced HD-map-based forecasting and motivates why map features are expected to matter, which the paper tests."}],"review_version":1}