Scaling Laws of Motion Forecasting and Planning -- Technical Report
Reviewed by Pithpith:GKLJYD2Fopen to challenge →
read the original abstract
We study the empirical scaling laws of a family of encoder-decoder autoregressive transformer models on the task of joint motion forecasting and planning in the autonomous driving domain. Using a 500 thousand hours driving dataset, we demonstrate that, similar to language modeling, model performance improves as a power-law function of the total compute budget, and we observe a strong correlation between model training loss and model evaluation metrics. Most interestingly, closed-loop metrics also improve with scaling, which has important implications for the suitability of open-loop metrics for model development and hill climbing. We also study the optimal scaling of the number of transformer parameters and the training data size for a training compute-optimal model. We find that as the training compute budget grows, optimal scaling requires increasing the model size 1.5x as fast as the dataset size. We also study inference-time compute scaling, where we observe that sampling and clustering the output of smaller models makes them competitive with larger models, up to a crossover point beyond which a larger models becomes more inference-compute efficient. Overall, our experimental results demonstrate that optimizing the training and inference-time scaling properties of motion forecasting and planning models is a key lever for improving their performance to address a wide variety of driving scenarios. Finally, we briefly study the utility of training on general logged driving data of other agents to improve the performance of the ego-agent, an important research area to address the scarcity of robotics data for large capacity models training.
This paper has not been read by Pith yet.
Forward citations
Cited by 14 Pith papers
-
Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Training
AutoScale is a closed-loop data engine using Graph-RAE for scene representation and Cluster-GA for importance-based retrieval to improve real-synthetic co-training for autonomous driving.
-
World Engine: Towards the Era of Post-Training for Autonomous Driving
World Engine generates realistic safety-critical driving variations from logs for reinforcement post-training, reducing benchmark failures more than data scaling and showing collision reductions plus on-road gains in ...
-
Scaling Self-Play for End-to-End Driving
Self-play DAgger training in a batched pixel renderer produces end-to-end driving policies that reach competitive performance on HUGSIM and NAVSIM-v2 after real-world adaptation and improve with more self-play compute.
-
STELLAR: Scaling 3D Perception Large Models for Autonomous Driving
STELLAR trains up to 500M-parameter multi-modal models on 50M driving scenes and reports empirical scaling trends plus new state-of-the-art results on the Waymo Open Dataset.
-
VL-DPO: Vision-Language-Guided Finetuning for Preference-Aligned Autonomous Driving
VL-DPO uses a VLM as a zero-shot reasoner to generate preference pairs from pretrained model rollouts, then finetunes via DPO on the Waymo Open End-to-End Driving Dataset, yielding 11.94% higher rater feedback score a...
-
Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems
MOSAIC is a scaling-aware data selection framework that outperforms baselines in training end-to-end autonomous driving planners, achieving comparable or better EPDMS scores with up to 80% less data.
-
SimScale: Learning to Drive via Real-World Simulation at Scale
SimScale synthesizes unseen driving states from real logs via neural rendering and reactive environments, generates pseudo-expert trajectories, and shows that co-training on real plus simulated data improves planning ...
-
DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
DriveVLA-W0 adds world modeling to predict future images in VLA models, overcoming sparse action supervision and amplifying data scaling laws on NAVSIM benchmarks and a large in-house dataset.
-
DeepFleet: Multi-Agent Foundation Models for Mobile Robots
DeepFleet develops and compares four foundation model architectures for multi-agent robot fleet coordination using warehouse data, finding robot-centric and graph-floor models most promising for prediction and scaling.
-
What Probing Reveals about Autonomous Driving: Linking Internal Prediction Errors to Ego Planning
Autonomous driving policies with strong closed-loop performance frequently lack timely internal predictions of surrounding vehicles during near-collision events, and causal correction of prediction errors leads to imp...
-
Rethinking Training & Inference for Forecasting: Linking Winner-Take-All back to GMMs
Links WTA training mismatch in GMM-modeled forecasters to uninformative posteriors and introduces post-hoc merging plus one-step EM to yield better-ranked mode probabilities without retraining.
-
Human-like autonomy emerges from self-play and a pinch of human data
Self-play RL regularized with 30 minutes of human data produces driving policies that coordinate with humans, training in 15 hours on one GPU with 2500x less data than imitation learning.
-
Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank
A differentiable motion forecasting model retrieves and refines interpretable trajectory anchors from a contrastively learned motion bank to improve transparency without sacrificing multi-modal accuracy.
-
Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive
Cross-benchmark analysis of 8 methods shows NAVSIM PDM Score correlates with Bench2Drive Driving Score at Spearman ρ=0.90, with Ego Progress as the strongest single predictor and a simpler 3-metric formula matching th...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.