FEABench: Evaluating Language Models on Multiphysics Reasoning Ability
read the original abstract
Building precise simulations of the real world and invoking numerical solvers to answer quantitative problems is an essential requirement in engineering and science. We present FEABench, a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA). We introduce a comprehensive evaluation scheme to investigate the ability of LLMs to solve these problems end-to-end by reasoning over natural language problem descriptions and operating COMSOL Multiphysics$^\circledR$, an FEA software, to compute the answers. We additionally design a language model agent equipped with the ability to interact with the software through its Application Programming Interface (API), examine its outputs and use tools to improve its solutions over multiple iterations. Our best performing strategy generates executable API calls 88% of the time. LLMs that can successfully interact with and operate FEA software to solve problems such as those in our benchmark would push the frontiers of automation in engineering. Acquiring this capability would augment LLMs' reasoning skills with the precision of numerical solvers and advance the development of autonomous systems that can tackle complex problems in the real world. The code is available at https://github.com/google/feabench
This paper has not been read by Pith yet.
Forward citations
Cited by 11 Pith papers
-
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
SCICONVBENCH is a new benchmark evaluating LLMs on multi-turn disambiguation and inconsistency resolution for task formulation in computational science, with frontier models reaching only 52.7% success on fluid mechan...
-
EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving
EngiBench shows LLMs accuracy drops with task complexity, degrades under perturbations, and stays below human performance on open-ended engineering problems.
-
A Constrained Natural-Language Interface for Variational Multi-Physics Finite Element Simulations in FEniCS
A constrained LLM front-end for FEniCS multi-physics simulations dispatches to human-written templates and achieves 100% valid parses plus 90-100% geometry success on benchmarks while avoiding LLM-generated solver code.
-
PDE-Agents: An LLM-Orchestrated Multi-Agent Framework for Automated Finite Element Simulations with Knowledge Graph-Augmented Reasoning
PDE-Agents shows a LangGraph-orchestrated multi-agent LLM framework with GraphRAG that reaches 100% task success and perfect material fidelity on novel materials in ablation tests, with 97.8% success across 1369 produ...
-
VFEAgent: A Multimodal Agent Framework for End-to-End Automated Finite Element Analysis
VFEAgent is an end-to-end multi-agent framework that automates FEA modeling and simulation from multimodal inputs, achieving high success rates in generating physically valid simulations across engineering scenarios.
-
Agentic AI for Particle-Based Simulation: Automating SPH Workflows for Debris Flow Modeling
An agentic AI workflow automates end-to-end SPH debris flow simulations via tool orchestration, multimodal inputs, and human-in-the-loop, demonstrating viability for meshless computational mechanics.
-
SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning
SciResearcher automates creation of diverse scientific reasoning tasks from academic evidence to train an 8B model that sets new SOTA at 19.46% on HLE-Bio/Chem-Gold and gains 13-15% on SuperGPQA-Hard-Biology and TRQA-...
-
Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPS
LLM syntax accuracy for LAMMPS scripts improved to 91% parser pass rate, yet only 1/80 scripts were scientifically correct on the hardest prompt; an agentic verification skill raised success to 5/6.
-
CAX-Agent: A Lightweight Agent Harness for Reliable APDL Automation
CAX-Agent is a three-layer agent harness for MAPDL automation whose model-driven recovery policy reaches 0.93 task completion and 0.84 zero-intervention rate on 50 simple structural benchmarks, outperforming rule-only...
-
SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning
SciResearcher is a new agentic data-construction framework that trains an 8B model via supervised fine-tuning and reinforcement learning to reach 19.46% on HLE-Bio/Chem-Gold and 13-15% gains on related biology and lit...
-
Enhancing Large Language Model-Based Systems for End-to-End Circuit Analysis Problem Solving
Hybrid pipeline using YOLO vision and ngspice verification raises circuit analysis accuracy from Gemini's 79.52% baseline to 97.59%, with similar gains on hand-drawn diagrams.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.