MetaOpenFOAM 2.0: Large Language Model Driven Chain of Thought for Automating CFD Simulation and Post-Processing

Hua Zhou; Xu Zhu; Yuxuan Chen; Zhuyin Ren

REVIEW 4 major objections 5 minor 3 references

MetaOpenFOAM 2.0: Large Language Model Driven Chain of Thought for Automating CFD Simulation and Post-Processing

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM agents automate CFD simulation and post-processing with an 86.9% pass rate.

desk verdict A useful engineering paper on LLM-driven CFD automation held back by an unverified human-judgment pass metric; the framework is plausible and the ablation is informative, but the headline Pass@1 needs a documented human evaluation before it stands. read the letter →

arxiv 2502.00498 v1 pith:DKTJFNTW submitted 2025-02-01 cs.AI physics.comp-ph

Yuxuan Chen , Xu Zhu , Hua Zhou , Zhuyin Ren This is my paper · ORCID

classification cs.AIphysics.comp-ph

keywords largelanguagemodelschainofthoughtmulti-agentframeworkCFDsimulationOpenFOAMpost-processingautomationretrieval-augmentedgenerationscalinglaw

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a multi-agent large language model framework, using chain-of-thought task decomposition and iterative verify-and-refine loops, can turn a plain-English simulation request into working OpenFOAM input files, run commands, and post-processing scripts. On a new 13-task benchmark covering fluid flow, heat transfer, combustion, visualization, and data extraction, it reports an average executability of 6.3 out of 7 and a Pass@1 rate of 86.9%, at an average cost of $0.15 per case. The baseline pipeline without the new decomposition and per-step review scored 2.1/7 and passed 0% of the tasks. If these results hold, a substantial part of the CFD setup and post-processing workflow, normally requiring specialist knowledge, can be delegated to an LLM-driven agent pipeline.

What carries the argument

The machinery is a two-level Chain of Thought: QDCOT (question-decomposition COT) first splits the user's request into CFD simulation and CFD post-processing, then each into executable subtasks such as writing input files, running solver commands, and running Python post-processing scripts. Each subtask then enters ICOT (iterative COT), a loop of an InputWriter agent that produces or rewrites files, a Runner agent that executes them, and a Reviewer agent that reads error output and returns feedback; a final LLM-assisted verification checks physical plausibility, flow characteristics, and boundary-condition consistency before the task is complete. A retrieval-augmented database of OpenFOAM tutorial cases supplies similar prior cases to guide the generated files.

What would settle it

Run the benchmark again after removing all tutorial cases that resemble the requested tasks from the retrieval database; if executability and Pass@1 collapse toward the values of the no-decomposition baseline, the reported gains come from retrieving near-solutions rather than from the chain-of-thought reasoning itself.

Watch

Extended reading notes

Core claim

The central claim is that the apparent difficulty of automating CFD with an LLM is not the physics or the solver syntax but the absence of structured reasoning: a single decomposition-and-run pass writes incorrect or incomplete files, whereas splitting the workflow into simulation and post-processing, then into subtasks, and reviewing each subtask's output before moving on, yields near-flawless task completion. Evidence is the jump from MetaOpenFOAM 1.0 to 2.0: the same underlying solver and retrieval database, with added two-stage question decomposition, per-subtask iterative review, and a final LLM-assisted physical plausibility check. The resulting system scores 6.3/7 executability and 86.9% Pass@1 across the benchmark, with visualization tasks reaching 7.0 on some cases and the hardest turbulence case (homogeneous isotropic turbulence) scoring about 5.3-5.4.

Load-bearing premise

The load-bearing premise is that the retrieval database contains OpenFOAM tutorial cases similar enough to whatever the user asks for; the HIT task's low-similarity match (cyclone instead of boxTurb16) lowered executability to about 5.3, and the paper gives no evidence that the LLM can produce correct inputs from general knowledge when no similar case exists.

Editorial extensions

If this is right

Non-expert users can describe a CFD case in natural language and receive runnable simulation inputs plus post-processed plots or extracted values, without manually editing OpenFOAM dictionaries.
The same COT structure should extend to other CFD workflow stages that the paper lists but does not test—mesh generation, pre-processing, and initialization—with additional QDCOT/ICOT loops raising executability further.
Accuracy and cost scale together: increasing decomposition and verification steps improves executability but raises token usage, so users can trade cost for reliability in a predictable way.
The benchmark gives the community a reusable 13-task evaluation for LLM-driven CFD, including post-processing tasks that previous benchmarks omitted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The benchmark reuses one PitzDaily simulation for six of its thirteen tasks, so the diversity of simulation setups is narrower than the task count suggests; a broader benchmark would show how well the result generalizes beyond tutorial-derived cases.
A stronger validation would replace the LLM's final plausibility check with quantitative physical criteria such as residuals, y-plus bounds, and conservation errors, making success auditable without the model judging its own output.
The reported scaling trend suggests that raising the ten-iteration cap might push executability higher, but cost rises with it, and the current data cannot distinguish continued gains from saturation.
Because the framework relies on retrieval of similar tutorial cases, the near-term practical path is expanding the case library rather than trusting the LLM's general knowledge for novel configurations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This is a competent and readable engineering paper. The genuinely new piece is MetaOpenFOAM 2.0: multiple decomposed chains of thought (QDCOT) for splitting simulation vs post-processing and each subtask, plus iterative verification (ICOT) loops, applied to OpenFOAM. The companion benchmark of 13 tasks explicitly covering post-processing (extraction and visualization) is a useful addition, since the existing MetaOpenFOAM 1.0 benchmark only addressed simulation. The ablation study is well structured: removing Python-based post-processing, then command-based post-processing, then all reviewers shows a clear monotonic improvement in executability, and the token/iteration accounting lets you see the trade-off. The appendix examples of the LLM-assisted verification catching a zero yPlus value and an empty plot are concrete and convincing. Cost reporting at $0.15 per case is useful context. The code link is a plus.

The soft spots are real and concentrated in evaluation. The Executability scale is mostly objective (0–5), but level 6 is the LLM reviewing its own outputs, and level 7—the only level counted as passing—is defined as requiring human judgment. The paper never describes the human evaluation: no annotator count, no rubric instructions beyond the one sentence, no blinding, no inter-annotator agreement, no raw scores. That makes the headline Pass@1 of 86.9% not yet established; it could be the LLM assigning itself 7s. This is a load-bearing issue, not a footnote, because every pass rate in Table 1 depends on it. The authors need to document who assigned the 7s and how.

Second, the benchmark is built from the same OpenFOAM tutorials that populate the RAG database. That inflates success rates. The paper partly acknowledges this—the HIT case shows a low-similarity cyclone match dropping executability to about 5.3—but it does not quantify how much of the headline result is retrieval of near-solutions. A test with cases outside the tutorial set would clarify the actual generalization.

Minor: the 'scaling laws' are fitted to a handful of ablation points; calling them post-training scaling laws is an overstatement. Also no error bars or confidence intervals on the averages, despite 10 runs.

Who is this for? Anyone building LLM-driven engineering workflows, and anyone designing benchmarks for LLM-for-science tools. The framework itself is credible and the ablations are informative. A serious referee should engage, not desk reject, but the paper needs a documented human evaluation, a generalization test, and more careful scaling claims before acceptance.

Referee Report

4 major / 5 minor

Summary. The paper presents MetaOpenFOAM 2.0, a multi-agent LLM framework built on MetaGPT and OpenFOAM that combines question-decomposition chain-of-thought (QDCOT), per-subtask iterative verification/refinement (ICOT), and LLM-assisted verification to turn natural-language CFD requests into simulations and post-processing. The authors introduce a 13-task benchmark adapted from OpenFOAM tutorials, an Executability rubric (0–7), a Pass@k metric, and report an average Executability of 6.3/7, a Pass@1 rate of 86.9%, and an average cost of $0.15 per case. They also report ablations showing gains from decomposing post-processing and from adding reviewers, and they claim token-based scaling laws for QDCOT and ICOT.

Significance. If the headline numbers hold, this is a useful applied demonstration that an LLM-driven agentic pipeline can automate standard OpenFOAM simulation setup and post-processing, with ablation evidence and a public benchmark/code release as strengths. The paper is honest about a known retrieval failure in Appendix A.4, which is a credit. However, the evaluation's top score—the only score that counts as a pass in Pass@k—requires human judgment, and no human evaluation protocol is reported. In addition, the benchmark overlaps with the RAG database contents, so the generalization claim is currently not established. The scaling-law claim is based on very few configurations and conflates architectural change with token budget.

major comments (4)

[§3.3, Pass@k definition; Table 1] Pass@1 is defined as the proportion of samples with an executability score of 7, and score 7 is defined in §3.3 as requiring human judgment. The paper reports no human evaluation protocol: no annotator count, no rubric instructions, no blinding, no inter-annotator agreement, and no raw score sheets. Since score 6 is the LLM reviewing its own outputs, the 6-to-7 boundary is exactly where the pass rate is determined. Without documented independent human scoring, the reported 86.9% Pass@1 and the per-task pass rates in Table 1 are not established. Please report the full human-scoring procedure and raw scores, or redefine the headline metric using objective execution outcomes (levels 0–5).
[§2.1, §3.2, Appendix A.4] The benchmark tasks in §3.2 are adapted from OpenFOAM tutorials, and the RAG database described in §2.1 is built from OpenFOAM documentation and tutorial cases. This creates a potential circularity: retrieved near-solutions may account for much of the observed success. The only evidence addressing this is the single HIT/cyclone example in Appendix A.4. Please report, for every task and every run, the retrieved case and its similarity score, and include at least a few tasks outside the tutorial database to test generalization to user requests that are not represented in the retrieval corpus.
[§4.2, Figures 7–8] The claim of a 'scaling law' is not supported by the presented data. It is based on three configurations for QDCOT and four for ICOT, with no error bars, no statistical tests, and no confidence intervals. More importantly, each configuration changes the framework's modules (adding or removing whole post-processing/review steps), so the plotted relationship between token usage and executability is not the same object as a post-training scaling law, which fixes the task and scales compute. Please either provide many more configurations with repeated runs and uncertainty quantification, or reframe the claim as a qualitative trend rather than a scaling law.
[Table 1, §3.4] All reported metrics are averages over 10 runs, but the paper gives no standard deviations, confidence intervals, or per-run score distributions. For binomial Pass@1 with n=10, two tasks with pass rates of 70% and 100% have overlapping confidence intervals; without uncertainty, apparent task differences such as HIT at 5.3 versus Cavity at 7.0 cannot be interpreted. Adding distributions or confidence intervals is needed to support the comparative claims in Section 3.4.

minor comments (5)

[§4.1] The text uses 'QRCOT' in the ablation descriptions; this should be 'QDCOT' for consistency with the rest of the paper.
[§3.4, Figure 3] The phrases 'Figure 3 (up)' and 'Figure 3 (down)' should be replaced with 'top panel' and 'bottom panel' or explicit subfigure labels.
[§3.3, metric (B)] The cost metric lists token usage and iterations but does not state the GPT-4o pricing used to convert tokens to dollars; please add the per-token prices and any fixed overhead assumptions so that the $0.15/case figure is reproducible.
[Appendix A.3] The text references 'Table 3' as presenting post-processing results, but no Table 3 appears in the manuscript; please either include it or correct the reference.
[§4.1, Figure 7 caption] Please clarify that ICOT(i) denotes the number of remaining reviewers (0 to 3), not a power or an exponent, and make the notation in the running text match the caption exactly.

Circularity Check

2 steps flagged · score 4.0 of 10

Evaluative circularity: Executability level 6 is defined as the LLM's own review of its outputs, and the reported Pass@1 depends on an undocumented human-judgment level 7; the headline metrics therefore are not fully independent of the system being measured.

self definitional [Section 3.1 Setup; Section 3.3 Evaluation Metrics (Executability); Section 3.4 Main Results]
"GPT-4o (OpenAI, 2024a) was selected as the primary LLM ... A score of '6' means the LLM reviews the results generated by the Python script (e.g., contours, extracted values) and assesses their validity based on factors such as user requirements, physical accuracy, flow characteristics, numerical accuracy and boundary condition consistency; A score of '7' indicates flawless results, requiring human judgment to verify whether the results are physically accurate and meet all user requirements."

The headline 'Executability score of 6.3' is the average of scores assigned under a rubric whose level 6 is explicitly 'the LLM reviews the results ... and assesses their validity.' The same GPT-4o powers the Reviewer and verification agents that generate the files and judge them, so the 6-level performance measure reduces to the system evaluating its own outputs. Since no independent scorer is described for the automated levels, the 6.3 average and the ablation gains built on it partly measure self-approval rather than externally verified correctness.
other [Section 3.3 Pass@k definition; Section 3.4 Main Results]
"c denotes the number of these samples that pass the test, i.e., achieve an executability score of 7. ... MetaOpenFOAM 2.0 demonstrates strong performance across 13 CFD simulation and post-processing tasks, achieving an average Executability score of 6.3 and a Pass@1 rate of 86.9%."

The pass definition itself invokes human judgment, but the paper reports no human evaluation protocol: no annotator count, instructions, blinding, inter-annotator agreement, or raw score sheets. Scores 0-6 are producible by the LLM pipeline, so the only documented boundary between self-assessment and independent verification is the score-7 level, and the paper does not show that any human assigned it. The reported Pass@1 of 86.9% therefore is not yet distinguished from the LLM's own level-6 approval in the evidence presented.

full rationale

No step in the CFD derivation chain reduces a predicted quantity to a fitted input: the benchmark cases are OpenFOAM tutorials run as executables, scores 0-5 are objective execution outcomes, and score 7 is defined as human judgment. The scaling-law claims are explicitly descriptive curve fits ('the lines fitted to these data points'), not extrapolative predictions. Self-citations to MetaOpenFOAM 1.0 are comparative baselines and are not load-bearing for the new framework's claims. The circularity concern is confined to the evaluation layer: level 6 of the Executability scale is defined as the LLM's own review of its outputs, and the reported Pass@1 depends on an undocumented human judgment at level 7. Because objective execution outcomes anchor the lower levels and the pass definition nominally requires a human, the central result is not forced by construction; the missing protocol, however, makes the headline pass rate unverified. Score 4 reflects partial evaluative circularity, not full reduction of the derivation to its inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the RAG database containing near-solutions for the benchmark tasks, on GPT-4o's ability to write correct OpenFOAM inputs, and on the validity of the self-designed evaluation metric. These are domain assumptions, not formal axioms, and they are not independently verified beyond the paper's own experiments.

free parameters (2)

temperature = 0.01
Set to 0.01 to reduce randomness in LLM output; affects reproducibility and performance, but is a hyperparameter, not fitted to data.
maximum iterations per ICOT = 10
Iteration cap used in the framework (Section 3.4); tasks reaching this cap have lower executability, indicating it influences results.

assumptions (3)

domain assumption The OpenFOAM documentation and tutorial cases in the RAG database are representative of general CFD simulation and post-processing tasks.
The benchmark tasks are adapted from OpenFOAM tutorials (Section 3.2); the system's performance depends on the relevance of retrieved cases (Appendix A.4 shows low-similarity matches reduce executability).
domain assumption GPT-4o can generate syntactically and semantically valid OpenFOAM input files and post-processing scripts when given decomposed subtasks and iterative feedback.
The entire framework hinges on the LLM's code generation ability; no formal guarantee is provided, and failures occur (e.g., HIT tasks have lower executability).
domain assumption The evaluation rubric (Executability 0-7) is a valid measure of task success.
The metric is introduced in this paper; score 6 is self-assessed by the LLM, so the metric's validity is not independently established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MetaOpenFOAM 2.0: Large Language Model Driven Chain of Thought for Automating CFD Simulation and Post-Processing." pith.science (2026). https://pith.science/paper/DKTJFNTW

@misc{pith2026250200498,
  author       = {Pith},
  title        = {Pith review of: MetaOpenFOAM 2.0: Large Language Model Driven Chain of Thought for Automating CFD Simulation and Post-Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DKTJFNTW}},
  note         = {Machine review of arXiv:2502.00498}
}

abstract

Computational Fluid Dynamics (CFD) is widely used in aerospace, energy, and biology to model fluid flow, heat transfer, and chemical reactions. While Large Language Models (LLMs) have transformed various domains, their application in CFD remains limited, particularly for complex tasks like post-processing. To bridge this gap, we introduce MetaOpenFOAM 2.0, which leverages Chain of Thought (COT) decomposition and iterative verification to enhance accessibility for non-expert users through natural language inputs. Tested on a new benchmark covering simulation (fluid flow, heat transfer, combustion) and post-processing (extraction, visualization), MetaOpenFOAM 2.0 achieved an Executability score of 6.3/7 and a pass rate of 86.9%, significantly outperforming MetaOpenFOAM 1.0 (2.1/7, 0%). Additionally, it proved cost-efficient, averaging $0.15 per case. An ablation study confirmed that COT-driven decomposition and iterative refinement substantially improved task performance. Furthermore, scaling laws showed that increasing COT steps enhanced accuracy while raising token usage, aligning with LLM post-training scaling trends. These results highlight the transformative potential of LLMs in automating CFD workflows for industrial and research applications. Code is available at https://github.com/Terry-cyx/MetaOpenFOAM

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages

[1]

AI, M. (2024). Introducing Llama 3.1: Our most capable models to date . https://ai.meta.com/blog/meta-llama-3-1/ Akata, E., Schulz, L., Coda-Forno, J., Oh, S. J., Bethge, M., & Schulz, E. (2023). Playing repeated games with large language models. arXiv preprint arXiv:2305.16867. An, J., Wang, H., Liu, B., Luo, K. H., Qin, F., & He, G. Q. (2020). A deep le...

arXiv 2024
[32]

OpenAI. (2024a). Hello gpt -4o. https://openai.com/index/hello-gpt-4o/ OpenAI. (2024b). Introducing OpenAI o1 . https://openai.com/o1/ Paul, D., West, R., Bosselut, A., & Faltings, B. (2024). Making Reasoning Matter: Measuring and Improving Faithfulness of Chain -of-Thought Reasoning. arXiv preprint arXiv:2402.13950. Wang, L., Ma, C., Feng, X., Zhang, Z.,...

arXiv 2024
[67]

J., & Chen, Z

Mao, R., Lin, M., Zhang, Y ., Zhang, T., Xu, Z.-Q. J., & Chen, Z. X. (2023). DeepFlame: A deep learning empowered open-source platform for reacting flow simulations. Computer Physics Communications , 291, 108842. Multiphysics, C. (1998). Introduction to comsol multiphysics®. COMSOL Multiphysics, Burlington, MA, accessed Feb, 9(2018),

work page 2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

AI, M. (2024). Introducing Llama 3.1: Our most capable models to date . https://ai.meta.com/blog/meta-llama-3-1/ Akata, E., Schulz, L., Coda-Forno, J., Oh, S. J., Bethge, M., & Schulz, E. (2023). Playing repeated games with large language models. arXiv preprint arXiv:2305.16867. An, J., Wang, H., Liu, B., Luo, K. H., Qin, F., & He, G. Q. (2020). A deep le...

arXiv 2024

[2] [32]

OpenAI. (2024a). Hello gpt -4o. https://openai.com/index/hello-gpt-4o/ OpenAI. (2024b). Introducing OpenAI o1 . https://openai.com/o1/ Paul, D., West, R., Bosselut, A., & Faltings, B. (2024). Making Reasoning Matter: Measuring and Improving Faithfulness of Chain -of-Thought Reasoning. arXiv preprint arXiv:2402.13950. Wang, L., Ma, C., Feng, X., Zhang, Z.,...

arXiv 2024

[3] [67]

J., & Chen, Z

Mao, R., Lin, M., Zhang, Y ., Zhang, T., Xu, Z.-Q. J., & Chen, Z. X. (2023). DeepFlame: A deep learning empowered open-source platform for reacting flow simulations. Computer Physics Communications , 291, 108842. Multiphysics, C. (1998). Introduction to comsol multiphysics®. COMSOL Multiphysics, Burlington, MA, accessed Feb, 9(2018),

work page 2023