{"id":"958ab200-9805-42cd-ac5a-eb0a4ece3194","arxiv_id":"2506.08516","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A retrospective of the ML4CFD competition shows a Gaussian-process-based entry outranking deep learning models and the OpenFOAM solver on a tailored multi-criteria score.","lead":"This paper reports the results of the NeurIPS 2024 ML4CFD competition, where over 240 teams built machine learning surrogates that predict air flow around 2D airfoils from simulation data. The winning entry, a classical Gaussian process method, outscored both deep learning models and the OpenFOAM solver on the competition's combined score, and the paper analyzes what made the top submissions work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Final-edition scoring thresholds are undisclosed, so the MMGP-vs-OpenFOAM margin (84.68 vs 82.5) cannot be audited or shown robust to threshold choice.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the undisclosed final-edition thresholds and the absence of sensitivity analysis over the scoring function. My independent read of the paper confirms this is the most critical point. The descriptive results (240 teams, 650 submissions, winning methods, public repositories) are internally consistent and plausible; the paper also candidly discloses known limitations such as the public test-set overfitting risk and the inclusion of evaluation time in the speed-up metric. I do not see an internal contradiction that would force rejection. However, the headline claim is a numerical ordering under a hand-designed scoring rule, and the scoring rule's key parameters are absent. The margin separating rank 1 from the OpenFOAM baseline is 2.18 points, which is well within the range of movement that threshold perturbations can produce. The malformed Table 1 and the conflicting speed-up figures (2-3x vs 300-600x vs 7000x in Sections 3, 4.2, and 6) are presentation-level defects that should be fixed, but they do not change the core issue: without the final thresholds and a sensitivity analysis, the central 'beats OpenFOAM' claim is not auditable. A CONDITIONAL verdict is therefore appropriate, and my stress-test does not move the verdict.","tokens_in":26730,"tokens_out":5083,"duration_ms":62123,"concrete_test":"Recover the exact final-edition thresholds and weights from the public LIPS/Codabench evaluation script or by asking organizers for T1/T2 per metric and the alpha values. Then, using the public AirfRANS test data and released MMGP code, recompute every metric entering Eq. (6) and sweep each threshold independently by +/-20% while varying the category weights over a plausible range (e.g., each of alpha_ML, alpha_OOD, alpha_PH in [0.2, 0.5]). If MMGP's score never falls below OpenFOAM's 82.5 under any feasible perturbation, the ranking is robust; if some perturbation flips the ordering, the paper must report this sensitivity before the headline claim can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that Safran-Tech/MMGP scored 84.68 while OpenFOAM scored 82.5 (Table 1, Section 3). The global score is a weighted sum (Eq. 6) of sub-scores obtained by binning each metric at two 'calibrated' thresholds into 0/1/2 points (Appendix B, Eq. 8), with weights alpha_ML=0.4, alpha_OOD=0.3, alpha_PH=0.3, alpha_A=0.75, alpha_S=0.25. The paper never reports the final-edition threshold values; the worked example in Appendix B is explicitly for the 'preliminary edition' and states that the criteria 'slightly evolved' for the new edition. Under the same scheme, OpenFOAM receives a fixed 82.5 (perfect non-speed sub-scores, zero speed-up score), so MMGP's margin is only 2.18 points. A small change in one threshold can move several metrics across a bin boundary and shift the global score by more than that margin. The only robustness check reported (Appendix D.1, speed-up inclusive/exclusive of evaluation time) does not stress thresholds or weights. Therefore the headline claim that a surrogate outperformed the reference solver is not independently auditable from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the results and retrospective analysis of the NeurIPS 2024 ML4CFD competition, which tasked participants with building surrogate models for steady incompressible RANS simulations of 2D airfoils from the AirfRANS dataset. The evaluation used the LIPS framework, combining ML accuracy and speed-up, physics compliance of lift and drag, and out-of-distribution generalization, aggregated into a single weighted global score. The paper describes the dataset and competition logistics, presents a leaderboard, and analyzes the four winning methods: MMGP (PCA with Gaussian processes and mesh morphing), OB-GNN (offset-based graph convolutions), MARIO (conditional neural fields), and GeoMPNN (geometry-aware message passing). The headline claim is that the MMGP entry achieved the highest global score (84.68), slightly above the OpenFOAM reference (82.5), and the authors draw design principles, particularly the importance of geometric inductive biases.","tokens_in":26956,"tokens_out":4918,"duration_ms":55484,"significance":"If the ranking is robust, this is a useful empirical benchmark: a non-deep-learning surrogate built on 103 scarce samples outperforms both deep learning surrogates and the reference CFD solver on a multi-criteria score. The paper has concrete strengths: it is based on a real competition with over 240 teams, the dataset is public, code links for the winning methods are provided, and the authors are transparent about several evaluation choices, including the definition of speed-up and training-cost issues (Appendix D.1). However, the central quantitative claim rests on an undisclosed threshold configuration for the scoring function, and there are internal inconsistencies in reported speed-up values. These issues currently limit the verifiability of the paper's main conclusions.","major_comments":[{"comment":"The final-edition thresholds T1 and T2 that convert each metric into 0, 1, or 2 points are not reported. Eq. (8) and the worked example in Table 3 apply to the preliminary edition, and the text explicitly states that the criteria 'slightly evolved' for the new edition. Because the global score (Eq. 6) is a weighted sum of these binned sub-scores, the leaderboard in Table 4 cannot be recomputed from the manuscript. The margin between the top entry (Safran-Tech, 84.68) and OpenFOAM (82.5) is only 2.18 points, so a small shift in thresholds could change several metrics' category memberships and potentially invert the headline ordering. The only robustness check reported (Appendix D.1) varies the speed-up definition, not the thresholds or the weights. Please disclose the final T1/T2 values for every metric and provide a sensitivity analysis over thresholds and weights demonstrating that the MMGP-versus-OpenFOAM ordering and the top-four ordering are stable.","section":"Appendix B / Table 4 / Eq. (8)"},{"comment":"The paper reports mutually inconsistent speed-up values for the same deep learning methods. Section 4.2 states that OB-GNN 'achieves a 7,000× speedup (220 ms vs 25 min per case),' Section 6 states that OB-GNN, MARIO, and GeoMPNN showed 'substantial computational speedups ranging from 300× to 600×,' and Section 3 states that the deep learning methods were 'typically 2–3× faster than MMGP.' These claims differ by orders of magnitude and cannot all describe the same evaluation. A single, consistent speed-up definition—preferably the competition metric in Eq. (11), which includes evaluation time—should be used throughout, and the reported numbers must be reconciled, since the paper's conclusions about the accuracy-speed trade-off depend on the correct order of magnitude.","section":"§4.2 / §6 / §3"}],"minor_comments":[{"comment":"There is a typo: 'everall score' should be 'overall score' in the Score Calculation Example.","section":"Appendix B"},{"comment":"There is a typo: 'seepdup' should be 'speedup'.","section":"Appendix E.5"},{"comment":"There is a typo: 'distrbutions' should be 'distributions'.","section":"§4.3"},{"comment":"There is a typo: 'ont of' should be 'one of'.","section":"Appendix E.2"},{"comment":"There is a typo: 'dimensionaly' should be 'dimensionality'.","section":"Appendix E.3"},{"comment":"The sub-scores in Table 1 (e.g., '162.7 162.8' for MMGP) are not defined; Table 4 reports the same entry's ML/Physics/OOD sub-scores as 0.89/0.88/0.76. Please clarify the scaling and column headers so the two tables are consistent and interpretable.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a competition retrospective rather than a methodological contribution; its value depends on the trustworthiness of the reported ranking. The undisclosed thresholds are a serious reproducibility issue for a paper whose main claim is a ranking, and the speed-up inconsistencies further weaken the empirical narrative. If the authors can release the final scoring configuration and a sensitivity analysis, the paper would be a solid benchmark report, but as written the headline claim is not fully auditable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the ML4CFD retrospective. The thing to know: it's a well-organized write-up of a real competition, but the central claim—that a mesh-morphing Gaussian process (MMGP) beat OpenFOAM on the aggregate score—can't be verified from the manuscript. The global score is a thresholded binning of each metric with weights given, but the actual thresholds T1 and T2 for the final edition are not reported. Appendix B gives a worked example for the 'preliminary edition' and notes the criteria 'slightly evolved.' So the 84.68 vs 82.5 margin is just 2.18 points, and a small threshold change could move several bins and flip the ranking. There's no sensitivity analysis over thresholds or weights, only a robustness check on whether to include evaluation time in the speed-up. That's a load-bearing gap.\n\nWhat's good: the paper is a candid retrospective, not a promotional piece. It reports 240 teams, 650 submissions, describes the four winning methods in detail (and their own authors wrote those sections, which is fine but should be flagged), and the competition code and data are public. The finding that a classical GP pipeline trained on 103 samples beats deep learning on this benchmark is genuinely interesting and worth discussing. The limitations about public test data and training costs are acknowledged openly.\n\nThe soft spots are addressable. The speed-up numbers are inconsistent: OB-GNN's own section claims 7,000×, the conclusion says 300–600× for deep learning, and Table 1 shows something else. That needs reconciliation. Table 1 is also hard to read. And there's no conflict-of-interest note for the winning teams' descriptions, though the affiliations make it obvious.\n\nThis is a paper for the ML-for-physics community and for anyone organizing SciML competitions. It deserves a serious referee, but only with required changes: report the thresholds, run a sensitivity analysis, fix the speed-up inconsistency, and clean up the tables. If those are addressed, it would be a solid citable benchmark report. As it stands, I'd take the descriptive parts at face value but wouldn't rely on the solver-beating claim.\n\nBest,","headline":"A useful competition retrospective whose headline claim is currently unauditable because the final scoring thresholds are never disclosed.","tokens_in":27647,"tokens_out":2686,"would_cite":true,"duration_ms":30499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The ML4CFD competition's top entry, a mesh-morphing Gaussian process, scored above the reference OpenFOAM solver on the aggregate evaluation.","keywords":["ML4CFD competition","surrogate modeling","computational fluid dynamics","mesh morphing Gaussian process","evaluation framework","out-of-distribution generalization","physics compliance"],"falsifier":"Recompute the global score for the top submissions using several plausible threshold sets (for example, varying the reported T1/T2 values by ±20 percent) and check whether MMGP still outranks OpenFOAM and the other finalists; if the rank order changes, the claim that a surrogate outperformed the reference solver is an artifact of the chosen thresholds.","tokens_in":26430,"feed_emoji":"🛩️","tokens_out":4336,"duration_ms":47082,"temperature":0.7,"pith_summary":"This retrospective analysis of the ML4CFD competition argues that a mesh-morphing Gaussian process (MMGP), trained on only 103 airfoil simulations, outperformed the reference OpenFOAM solver on the competition's aggregate score, which combines predictive accuracy, speed-up, physical compliance of lift and drag, and out-of-distribution generalization. The paper claims this shows classical, non-deep-learning surrogates can remain competitive when geometry is reparametrized well, while deep graph and neural-field models offer much larger speedups. It also distills design principles for future scientific ML competitions: geometric inductive biases, careful scoring of evaluation time, multiple baselines, and statistical re-evaluation of final rankings.","feed_headline":"Mesh-morphing GP outscored OpenFOAM in ML4CFD","feed_subtitle":"A 103-sample Gaussian-process pipeline outscored deep nets and the CFD solver on accuracy, physics, and OOD scores.","key_machinery":"The load-bearing object is the global score, a linear combination of three subscores (ML accuracy and speed-up, physics compliance, OOD generalization) in which each metric is binned at two thresholds into 0, 1, or 2 points before being averaged; the weights are $\\alpha_{ML}=0.4$, $\\alpha_{OOD}=0.3$, $\\alpha_{PH}=0.3$, with accuracy weighted at 0.75 against speed-up at 0.25. This scoring function is what lets the paper claim a surrogate can 'outperform' a high-fidelity solver, so the binned-threshold design carries the entire ranking argument. On the method side, the MMGP pipeline carries the winning result: morphing aligns each mesh's wake, snapshot POD compresses fields, and independent Gaussian processes regress the generalized coordinates, with the zero-velocity boundary condition preserved automatically by the linear reconstruction.","core_discovery":"The paper's central claim is that the first-place solution, MMGP, achieved a global score of 84.68 that exceeds the OpenFOAM reference's score of 82.5 on a weighted combination of accuracy, speed-up, physics compliance of lift and drag, and out-of-distribution generalization (Table 4, rank 1, Safran-Tech). The authors attribute this result not to a larger model but to a deterministic reparametrization pipeline — mesh morphing onto a common shape, finite element interpolation onto a common mesh, snapshot POD dimensionality reduction, and independent Gaussian process regressors per generalized coordinate — which converts variable-size CFD meshes into fixed-size low-dimensional embeddings. They report that the winning approach beat both deep learning surrogates and the CFD solver on aggregate metrics, and that all top methods explicitly encoded geometric structure.","pith_inferences":["A natural robustness test is to recompute the leaderboard over the full threshold and weight space; the paper does not report sensitivity of the rank order beyond the speed-up metric, so the headline claim may be threshold-dependent.","The MMGP recipe — align geometries, compress, regress — could transfer to other non-parameterized geometric variability problems, such as patient-specific hemodynamics or turbomachinery.","The paper's reported speedup ratios (300–600x for neural methods versus MMGP) imply an accuracy/speed Pareto frontier; a future competition could score on the frontier rather than a fixed linear combination.","Combining MMGP's morphing with neural inference speed might yield a strictly dominant surrogate, a hybrid the authors mention only implicitly."],"forward_implications":["If the claim holds, classical kernel-based surrogates should be treated as serious baselines for small-data PDE problems, not just deep learning.","Geometric reparametrization (morphing) can matter more than architectural complexity when training data is scarce.","The multi-criteria evaluation framework, with explicit physics and OOD categories, provides a template for comparing ML surrogates against traditional solvers in other engineering domains.","Deep learning methods' larger speedups are under-rewarded by the scoring weights, so different weight choices could change which approach wins; the paper acknowledges this tradeoff.","Future SciML competitions should adopt the paper's recommendations: hidden test sets, multiple baselines, and statistical re-evaluation of final rankings."],"supporting_citations":[{"why":"Supplies the LIPS benchmarking framework used for the multi-criteria evaluation and ranking.","marker":"[14]"},{"why":"Provides the AirfRANS dataset, the CFD simulation data, and the task definition for airfoil surrogate modeling.","marker":"[16]"},{"why":"Describes the MMGP method that achieved the top global score in the competition.","marker":"[22]"},{"why":"Defines the competition setup and dataset splits used in the ML4CFD challenge.","marker":"[19]"},{"why":"Documents the preliminary edition's scoring parameters and leaderboard, used as the basis for the final scoring.","marker":"[33]"},{"why":"Provides the critique of weak baselines and reporting biases that motivates the paper's evaluation and benchmarking recommendations.","marker":"[32]"}],"fun_headline_variants":["Winning ML4CFD entry beats OpenFOAM with mesh morphing","Gaussian process with mesh morphing tops ML4CFD, beats OpenFOAM","ML4CFD champ: morphing GP outdoes OpenFOAM on aggregate metrics","How a morphing GP outperformed OpenFOAM in ML4CFD","Winning ML4CFD solution: morphing GP beats OpenFOAM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ranking — including the claim that the top entry beats OpenFOAM — rests on hand-chosen thresholds and weights in the scoring function, whose final-edition values are not reported and whose sensitivity is not tested; if a different reasonable threshold set changed category memberships, the leaderboard and its conclusions would change.","fun_headline_variants_meta":{"raw":{"variants":["Winning ML4CFD entry beats OpenFOAM with mesh morphing","Gaussian process with mesh morphing tops ML4CFD, beats OpenFOAM","ML4CFD champ: morphing GP outdoes OpenFOAM on aggregate metrics","How a morphing GP outperformed OpenFOAM in ML4CFD","Winning ML4CFD solution: morphing GP beats OpenFOAM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2728,"prompt_tokens":919,"completion_tokens":1809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1704}},"tokens_in":535,"tokens_out":1809,"duration_ms":14051,"temperature":1.0,"reasoning_tokens":1704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:09:57.060711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the global score for the top submissions using several plausible threshold sets (for example, varying the reported T1/T2 values by ±20 percent) and check whether MMGP still outranks OpenFOAM and the other finalists; if the rank order changes, the claim that a surrogate outperformed the reference solver is an artifact of the chosen thresholds.","supporting_citations":[{"cited_title":"LIPS-learning industrial physical simulation benchmark suite","cited_arxiv_id":null,"evidence_quote":"Supplies the LIPS benchmarking framework used for the multi-criteria evaluation and ranking."},{"cited_title":"Airfrans: High fi- delity computational fluid dynamics dataset for approximating reynolds-averaged navier–stokes solutions","cited_arxiv_id":null,"evidence_quote":"Provides the AirfRANS dataset, the CFD simulation data, and the task definition for airfoil surrogate modeling."},{"cited_title":"MMGP: a Mesh Morphing Gaus- sian Process-based machine learning method for regression of physical problems under non- parametrized geometrical variability","cited_arxiv_id":null,"evidence_quote":"Describes the MMGP method that achieved the top global score in the competition."},{"cited_title":"NeurIPS 2024 ML4CFD Competition: Harnessing Machine Learning for Computational Fluid Dynamics in Airfoil Design","cited_arxiv_id":"2407.01641","evidence_quote":"Defines the competition setup and dataset splits used in the ML4CFD challenge."},{"cited_title":"ML4PhySim : Machine Learning for Physical Simulations Challenge (The airfoil design)","cited_arxiv_id":"2403.01623","evidence_quote":"Documents the preliminary edition's scoring parameters and leaderboard, used as the basis for the final scoring."},{"cited_title":"Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations","cited_arxiv_id":null,"evidence_quote":"Provides the critique of weak baselines and reporting biases that motivates the paper's evaluation and benchmarking recommendations."}],"review_version":1}