REVIEW 4 major objections 6 minor 1 cited by
A Benchmarking Framework for AI models in Automotive Aerodynamics
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A unified benchmark for AI car-aerodynamics models compares three architectures on CFD-grade metrics, exposing which ones truly capture drag and lift trends.
desk verdict A genuinely useful open-source benchmarking framework for AI in automotive aerodynamics, but the paper's own model comparison is undermined by evaluating DoMINO on mesh points while the other two models are interpolated back from point clouds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the benchmarking suite itself: a collection of metrics and workflows that ingest model predictions on shared points (as .vtp surface and .vtu volume files) and compute area-weighted L2 errors, Spearman design-trend coefficients, R2 force regressions, centerline line plots, and physics-based conservation residuals. A uniform point-cloud utility samples points directly from STL files and integrates forces over them, decoupling evaluation from the simulation mesh. The framework also fixes a reproducible train-validation split of DrivAerML (90/10 with the extreme-drag samples forced into validation) so that any model can be compared on an equal footing.
What would settle it
Retrain X-MeshGraphNet and FIGConvNet with an integrated drag/lift loss term and evaluate them directly on mesh points instead of interpolated point clouds; if their area-weighted L2 errors and force regressions do not approach or exceed DoMINO's, the paper's architectural explanations are supported, whereas if they close the gap, the reported ranking was an artifact of the evaluation scheme.
Extended reading notes
Core claim
The paper's central claim is that a standardized and extensible evaluation workflow can produce consistent, physically meaningful comparisons of AI models for automotive aerodynamics, and that applying it to three representative models on the DrivAerML dataset exposes real architectural differences that conventional error metrics miss. Specifically, the authors report that DoMINO achieves the lowest area-weighted L2 errors for pressure and wall shear stress, and the best drag and lift regression (R2 of 0.98 and 0.97), because it trains directly on simulation mesh points and includes an integrated-quantity loss. FIGConvNet matches DoMINO on Spearman design-trend coefficients (0.99 drag, 0.98 lift) while X-MeshGraphNet lags on trends and regression despite comparable pointwise L2 errors, a gap the authors attribute to the absence of an integrated loss and to point-cloud interpolation during training.
Load-bearing premise
The comparisons are valid even though DoMINO trains and evaluates directly on simulation mesh points while X-MeshGraphNet and FIGConvNet are trained on interpolated uniform point clouds and their predictions are interpolated back to the mesh, so any systematic bias from that interpolation could change the reported error rankings.
Editorial extensions
If this is right
- New models added to PhysicsNeMo-CFD can be compared against these three on identical metrics, making DrivAerML a common reference point for automotive aerodynamics surrogates.
- Design-trend and Spearman metrics give engineers a quantitative check on whether a model can be trusted for what-if design studies, not just for average error.
- Uniform point-cloud evaluation removes meshing as a bottleneck and as a source of comparison inconsistency across models.
- Physics-based residual metrics reveal whether a model roughly satisfies conservation laws, guiding whether it can initialize or validate CFD solvers.
- Architecture choices such as integral loss terms and training on mesh points directly affect force and trend accuracy, giving model developers concrete guidance.
Reading between the lines
- The framework's volume comparisons currently cover only DoMINO; extending the same volume workflows to FIGConvNet and X-MeshGraphNet would test whether DoMINO's volume advantage is intrinsic or an artifact of limited baselines.
- The proposed train-validation split, which forces extreme-drag cases into validation, could serve as a template for other engineering datasets where generalization to out-of-distribution designs is the real goal.
- A shared canonical interpolation scheme, rather than each model choosing its own point-cloud and interpolation parameters, would remove a remaining confound in comparing any two models on surface and volume fields.
- The same point-cloud measurement protocol could be exported to other external aerodynamics problems, such as aircraft or bluff-body flows, wherever STL geometry and CFD ground truth are available.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents PhysicsNeMo-CFD, an open-source benchmarking framework for evaluating AI surrogate models for automotive aerodynamics. The framework standardizes evaluation along several axes: aerodynamic force regression and design-trend analysis (Spearman correlation, mean/max absolute errors), surface and volume field comparisons (L2 and area-weighted L2 errors, centerline plots, contours), physics-based residual computation, and validation on uniform surface point clouds. A fixed 90/10 train/validation split for the DrivAerML dataset is proposed. To demonstrate the framework, the authors benchmark three NVIDIA models—DoMINO, X-MeshGraphNet, and FIGConvNet—on 48 DrivAerML validation samples, reporting surface results for all three models and volume results for DoMINO only. The paper's central claim is that the framework provides a robust, standardized, and reproducible basis for comparing AI models from the perspective of the CAE community.
Significance. The framework addresses a real gap: CFD and ML communities currently report incomparable metrics, and PhysicsNeMo-CFD's requirement that predictions and ground truth be provided on the same points (Section 2.2.1), its area-weighted surface errors, design-trend statistics, and the public DrivAerML split are sensible and useful design choices; the code is released under an open-source repository, which is a concrete asset for the community. The quantitative results for the three in-house models are plausible and internally consistent with the stated training setups, and the point-cloud force evaluation (Tables 7-9) is a reasonable second-level check. However, the demonstration as presented does not yet establish that the framework yields fair inter-model comparisons: the main field-accuracy metric is computed on a representation that matches only one model's native output, volume benchmarking is demonstrated for a single model, and no uncertainty quantification accompanies the rankings. The contribution is therefore significant conditional on the demonstration being strengthened to a true common evaluation basis.
major comments (4)
- [Section 3.3, Tables 2-3] Section 3.3, Tables 2-3: the headline field-accuracy ranking is not computed on a common evaluation basis. DoMINO is trained and evaluated directly on simulation-mesh cell centers, while X-MeshGraphNet and FIGConvNet are trained on uniform point clouds (4.5M and 500K points, respectively) and their predictions are linearly interpolated back to the mesh for benchmarking; the L2 and area-weighted L2 errors on cell centers therefore include an interpolation step for two of the three models but not for DoMINO. The manuscript itself (Section 3.4.1) states that DoMINO's area-weighted advantage 'is likely because DoMINO trains on the surface mesh data ... directly,' which is exactly the confound. Because Section 2.2.1 requires that true and predicted results be supplied on the same points, the demonstration should report field L2 errors for all three models evaluated on a single common point set—for example, the 10M-point cloud already used for the force comparisons in Tables 7-9, with DoMINO queried on those same points. Until such a common-basis field comparison is reported, the relative ordering in Tables 2-3 cannot be attributed to model quality rather than to the evaluation pipeline.
- [Section 3.4, Tables 4-9] Section 3.4, Tables 4-9: the comparative claims (e.g., 'FIGConvNet and DoMINO perform better than X-MeshGraphNet') rest on single-point estimates of R2 and Spearman coefficients computed from 48 validation samples, with no error bars, confidence intervals, or per-sample distributions reported. On 48 samples the difference between Spearman 0.96 and 0.99, or between R2 Lift 0.95 and 0.97, is not obviously statistically meaningful, and the deliberate inclusion of low- and high-drag out-of-distribution samples means a few outliers could dominate the summary statistics. Please report bootstrap confidence intervals for the Spearman and R2 values, per-sample error distributions, and a separate breakdown of results for in-distribution versus out-of-distribution validation subsets.
- [Section 3.4.2, Table 10] Section 3.4.2, Table 10: volume benchmarking is demonstrated for DoMINO only; the paper states that X-MeshGraphNet's volume model was not fine-tuned and FIGConvNet's volume model is still in development. Since the framework's advertised scope includes volumetric flow-field prediction and standardized cross-model comparison, the present demonstration does not show that the volume suite supports inter-model comparison—only single-model reporting. Either add at least one additional volume model to the evaluation or state explicitly in Section 4 that volume benchmarking is currently validated for single-model use only, so that readers do not infer a capability the paper does not exhibit.
- [Section 4] Section 4: the utility demonstration carries a self-selection risk that the paper itself exposes when it states that the benchmarking framework 'was crucial throughout the development of these models,' and Sections 3.3–3.4 show that design choices such as training representation (mesh versus point cloud) and the presence of integral-loss terms differ across the three models in ways the framework's metrics reward. The three models are therefore not fully independent witnesses to the framework's value: DoMINO's area-weighted L2 advantage is attributed to a training-representation choice that the benchmarking framework presumably informed. As a concrete test of the framework's portability, please benchmark at least one model developed outside this group (or at least not tuned against PhysicsNeMo-CFD's metrics) and discuss whether the in-house comparison is best read as a property of the model families or of development choices already guided by the benchmark. At minimum, this limitation should be acknowledged where the comparative conclusions are drawn (Sections 3.4.1 and 4).
minor comments (6)
- [Throughout] Please correct typos: 'Simularly' (Section 2), 'the the benchmarking framework' (Section 3.4), 'V olume results' (Section 3.4.2 heading), 'The training and testing ids of consistent' (Section 3.3), and 'A verage error metrics' (Section 3.4.1).
- [Section 3.4.1, Tables 2-3] Define the L2 error metric precisely: state whether it is the normalized L2 (e.g., ||pred - truth||2 / ||truth||2), how it is aggregated over the 48 validation samples, how point-cloud predictions are interpolated onto cell centers, and what interpolation order is used, so that the tables are reproducible from the released code.
- [Section 2.1.3] Physics-based residual metrics are described as part of the framework, but Section 3 reports no residual results for any model; please either report residual-based quantities (e.g., mass-conservation residuals on the validation set) or state explicitly that residual evaluation is not exercised in this demonstration.
- [Section 2.2.1] The validation-split description is ambiguous: '10% of the validation set is chosen to be the top of the sorted dataset'—please clarify whether the top/bottom quotas are percentages of the validation set or of the full dataset, and specify the exact algorithm (and random seed) used to draw the middle 80% of the validation set.
- [References] The reference list mixes preprints with the software citation '[10] P. Contributors'; please provide a dedicated Code and Data Availability statement, including a versioned DOI or commit hash for the PhysicsNeMo-CFD repository, so that the exact evaluation code used for Tables 2-10 can be pinned.
- [Section 3.2.2] The claim that the workflows were 'tested' on DriveSim, DriveSim+, and an aircraft dataset is supported only by qualitative figures; please either add quantitative metrics for those datasets or state explicitly that those tests were limited to qualitative sanity checks, consistent with the sentence in Section 3.2.2 noting that only DrivAerML is fully integrated.
Circularity Check
No construction-level circularity: benchmark metrics are computed against external CFD ground truth; the acknowledged mesh-versus-point-cloud evaluation mismatch and same-group model development are fairness confounds, not definitional reductions.
full rationale
The paper's central claim is that PhysicsNeMo-CFD provides a reusable, extensible benchmarking framework with CFD-relevant metrics. This is not a derived physical result; the metrics (L2, area-weighted L2, R2, Spearman, residuals) are standard and are evaluated against held-out DrivAerML CFD ground truth rather than being defined from the models' outputs. The most plausible circularity candidate is the evaluation-representation difference in Section 3.3: DoMINO trains and evaluates on simulation mesh points while X-MeshGraphNet and FIGConvNet are trained on interpolated uniform point clouds and interpolated back to the mesh for the headline L2 tables. This is a real validity confound, and the paper itself says DoMINO's area-weighted advantage 'is likely because DoMINO trains on the surface mesh data directly'; but the reported errors still compare to external ground truth and are not equal to any fitted parameter by construction. The self-referential demonstration (Section 4 states the framework 'was crucial throughout the development of these models' for the same NVIDIA models used as the demonstration) reduces confidence in the model rankings but does not make the benchmark claim reduce to its input. Self-citations to the architecture papers describe the benchmarked models and are not load-bearing for the framework's validity as open-source, externally usable tooling. Accordingly, no circular step meeting the quote-and-reduction standard was found; the score reflects only the mild self-referential demonstration.
Assumptions & free parameters
assumptions (3)
- domain assumption DrivAerML ground truth is accurate and representative.
- domain assumption The selected metrics (L2, area-weighted L2, Spearman trend, R2) capture performance relevant to the CAE community.
- domain assumption Point-cloud sampling and linear interpolation between meshes and point clouds preserve the fidelity needed for fair comparison.
Cite this review
Pith. "Pith review of A Benchmarking Framework for AI models in Automotive Aerodynamics." pith.science (2026). https://pith.science/paper/3UKQ6CRX
@misc{pith2026250710747,
author = {Pith},
title = {Pith review of: A Benchmarking Framework for AI models in Automotive Aerodynamics},
year = {2026},
howpublished = {\url{https://pith.science/paper/3UKQ6CRX}},
note = {Machine review of arXiv:2507.10747}
}
read the original abstract
In this paper, we introduce a benchmarking framework within the open-source NVIDIA PhysicsNeMo-CFD framework designed to systematically assess the accuracy, performance, scalability, and generalization capabilities of AI models for automotive aerodynamics predictions. The open extensible framework enables incorporation of a diverse set of metrics relevant to the Computer-Aided Engineering (CAE) community. By providing a standardized methodology for comparing AI models, the framework enhances transparency and consistency in performance assessment, with the overarching goal of improving the understanding and development of these models to accelerate research and innovation in the field. To demonstrate its utility, the framework includes evaluation of both surface and volumetric flow field predictions on three AI models: DoMINO, X-MeshGraphNet, and FIGConvNet using the DrivAerML dataset. It also includes guidelines for integrating additional models and datasets, making it extensible for physically consistent metrics. This benchmarking study aims to enable researchers and industry professionals in selecting, refining, and advancing AI-driven aerodynamic modeling approaches, ultimately fostering the development of more efficient, accurate, and interpretable solutions in automotive aerodynamics
Figures
Figures from the paper (9 more)
Forward citations
Cited by 1 Pith paper
-
GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer
GeoTransolver, a geometry-aware attention transformer, improves surrogate CFD accuracy over existing baselines on three automotive/aerospace datasets, but the paper has major reporting gaps.
Reference graph
Works this paper leans on
-
[1]
B. Hupertz, N. Lewington, C. Mockett, N. Ashton, and L. Duan, “Towards a Standardized Assessment of Automotive Aerodynamic CFD Prediction Capability - AutoCFD 2: Ford DrivAer Test Case Summary,” inSAE Technical Papers, no. 2022, SAE International, 3 2022
work page 2022
-
[2]
N. Ashton, P. Unterlechner, and T. Blacha, “Assessing the Sensitivity of Hybrid RANS-LES Simulations to Mesh Resolution, Numerical Schemes and Turbulence Modelling within an Industrial CFD Process,” pp. 1–11, 4 2018
work page 2018
-
[3]
Hybrid les/rans methods for the simulation of turbulent flows,
J. Fr ¨ohlich and D. V on Terzi, “Hybrid les/rans methods for the simulation of turbulent flows,”Progress in Aerospace Sciences, vol. 44, no. 5, pp. 349–377, 2008
work page 2008
-
[4]
M. Elrefaie, F. Ahmed, and A. Dai, “Drivaernet: A parametric car dataset for data-driven aerodynamic design and graph-based drag prediction,” inInternational Design Engineering Technical Conferences and Computers and Information in Engineering Conference, vol. 88360, p. V03AT03A019, American Society of Mechanical Engineers, 2024
work page 2024
-
[5]
Aerodynamics-guided machine learning for design optimization of electric vehicles,
J. Tran, K. Fukami, K. Inada, D. Umehara, Y . Ono, K. Ogawa, and K. Taira, “Aerodynamics-guided machine learning for design optimization of electric vehicles,”Communications Engineering, vol. 3, no. 1, p. 174, 2024
work page 2024
-
[6]
Surrogate modeling of car drag coefficient with depth and normal renderings,
B. Song, C. Yuan, F. Permenter, N. Arechiga, and F. Ahmed, “Surrogate modeling of car drag coefficient with depth and normal renderings,” inInternational Design Engineering Technical Conferences and Com- puters and Information in Engineering Conference, vol. 87301, p. V03AT03A029, American Society of Mechanical Engineers, 2023
work page 2023
-
[7]
M. Elrefaie, F. Morar, A. Dai, and F. Ahmed, “Drivaernet++: A large-scale multimodal car dataset with computational fluid dynamics simulations and deep learning benchmarks,”Advances in Neural Informa- tion Processing Systems, vol. 37, pp. 499–536, 2024
work page 2024
-
[8]
WindsorML: High-Fidelity Computational Fluid Dynamics Dataset For Automotive Aerodynamics
N. Ashton, J. B. Angel, A. S. Ghate, G. K. Kenway, M. L. Wong, C. Kiris, A. Walle, D. C. Maddix, and G. Page, “Windsorml: High-fidelity computational fluid dynamics dataset for automotive aerodynamics,” arXiv preprint arXiv:2407.19320, 2024
work page Pith review arXiv 2024
Show all 21 references
-
[9]
Factorized implicit global convolution for automotive computational fluid dynamics prediction,
C. Choy, A. Kamenev, J. Kossaifi, M. Rietmann, J. Kautz, and K. Azizzadenesheli, “Factorized implicit global convolution for automotive computational fluid dynamics prediction,”arXiv preprint arXiv:2502.04317, 2025
2025 arXiv
-
[10]
NVIDIA PhysicsNeMo: An open-source framework for physics-based deep learn- ing in science and engineering,
P. Contributors, “NVIDIA PhysicsNeMo: An open-source framework for physics-based deep learn- ing in science and engineering,” 2023. [Online]. Available:https://github.com/NVIDIA/ physicsnemo
2023
-
[11]
3d flow field estimation around a vehicle using convolutional neural networks.,
F. Chen and K. Akasaka, “3d flow field estimation around a vehicle using convolutional neural networks.,” inBMVC, p. 396, 2021
2021
-
[12]
Deep learning for real-time aerodynamic evaluations of arbitrary vehicle shapes,
S. J. Jacob, M. Mrosek, C. Othmer, and H. K ¨ostler, “Deep learning for real-time aerodynamic evaluations of arbitrary vehicle shapes,”arXiv preprint arXiv:2108.05798, 2021
2021 arXiv
-
[13]
Domino: A decomposable multi-scale iterative neural operator for modeling large scale engineering simulations,
R. Ranade, M. A. Nabian, K. Tangsali, A. Kamenev, O. Hennigh, R. Cherukuri, and S. Choudhry, “Domino: A decomposable multi-scale iterative neural operator for modeling large scale engineering simulations,”arXiv preprint arXiv:2501.13350, 2025
2025 arXiv
-
[14]
X-meshgraphnet: Scalable multi-scale graph neural networks for physics simulation,
M. A. Nabian, C. Liu, R. Ranade, and S. Choudhry, “X-meshgraphnet: Scalable multi-scale graph neural networks for physics simulation,”arXiv preprint arXiv:2411.17164, 2024
2024 arXiv
-
[15]
3d super-resolution model for vehicle flow field enrich- ment,
T. L. Trinh, F. Chen, T. Nanri, and K. Akasaka, “3d super-resolution model for vehicle flow field enrich- ment,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5826– 5835, 2024
2024
-
[16]
Drag-guided diffusion models for vehicle image gen- eration,
N. Arechiga, F. Permenter, B. Song, and C. Yuan, “Drag-guided diffusion models for vehicle image gen- eration,”arXiv preprint arXiv:2306.09935, 2023. 23
2023 arXiv
-
[17]
Shapenet: An information-rich 3d model repository,
A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “Shapenet: An information-rich 3d model repository,” 2015
2015
-
[18]
Drivaerml: High-fidelity computational fluid dynamics dataset for road-car external aerodynamics,
N. Ashton, C. Mockett, M. Fuchs, L. Fliessbach, H. Hetmann, T. Knacke, N. Schonwald, V . Skaperdas, G. Fotiadis, A. Walle,et al., “Drivaerml: High-fidelity computational fluid dynamics dataset for road-car external aerodynamics,”arXiv preprint arXiv:2408.11969, 2024
2024 arXiv
-
[19]
Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators,
L. Lu, P. Jin, and G. E. Karniadakis, “Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators,”arXiv preprint arXiv:1910.03193, 2019
1910 arXiv
-
[20]
Ahmedml: High-fidelity computa- tional fluid dynamics dataset for incompressible, low-speed bluff body aerodynamics,
N. Ashton, D. C. Maddix, S. Gundry, and P. M. Shabestari, “Ahmedml: High-fidelity computa- tional fluid dynamics dataset for incompressible, low-speed bluff body aerodynamics,”arXiv preprint arXiv:2407.20801, 2024
2024 arXiv
-
[21]
Weatherbench 2: A benchmark for the next generation of data-driven global weather models,
S. Rasp, S. Hoyer, A. Merose, I. Langmore, P. Battaglia, T. Russel, A. Sanchez-Gonzalez, V . Yang, R. Carver, S. Agrawal, M. Chantry, Z. B. Bouallegue, P. Dueben, C. Bromberg, J. Sisk, L. Barrington, A. Bell, and F. Sha, “Weatherbench 2: A benchmark for the next generation of ...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.