BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction

Chenglei Yu; Long Wei; Tailin Wu; Tianrun Gao; Tian Xia; Wenhao Deng; Xiaowei Qian

arxiv: 2510.16559 · v5 · pith:IPYSRDS6new · submitted 2025-10-18 · 💻 cs.AI

BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction

Tian Xia , Tianrun Gao , Wenhao Deng , Long Wei , Xiaowei Qian , Chenglei Yu , Tailin Wu This is my paper

Pith reviewed 2026-05-21 20:10 UTC · model grok-4.3

classification 💻 cs.AI

keywords LLM benchmarkengineering constructionphysics-aligned evaluationlanguage-driven automation3D spatial computationstatic and dynamic mechanicsinteractive benchmarkconstruction automation

0 comments

The pith

BuildArena is the first benchmark that tests LLMs on turning language instructions into physically viable 3D structures.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces BuildArena to evaluate large language models on engineering construction tasks that demand integrated reasoning under physical constraints. It fills an evaluation gap by creating interactive scenarios where models must produce constructions that respect static and dynamic mechanics. A sympathetic reader would care because progress here could move LLMs from abstract planning toward practical automation in domains where physical feasibility determines success or failure. The benchmark includes tasks across difficulty levels and supplies supporting computational tools for 3D geometry. Evaluation results on nine frontier models provide the first systematic view of current LLM strengths and limits in this setting.

Core claim

BuildArena is the first physics-aligned interactive benchmark designed for language-driven engineering construction. It takes a first step towards engineering automation using LLMs through an extendable task design strategy spanning static and dynamic mechanics across multiple difficulty tiers and a 3D Spatial Geometric Computation Library for supporting construction based on language instructions. On nine frontier LLMs, BuildArena comprehensively evaluates their capabilities for language-driven and physics-grounded construction automation.

What carries the argument

The extendable task design strategy spanning static and dynamic mechanics together with the 3D Spatial Geometric Computation Library, which together generate tasks and verify outputs against physical constraints from language input.

If this is right

LLMs can now be ranked on their ability to handle both static stability and dynamic motion in construction sequences.
Performance gaps across difficulty tiers will highlight which types of physical reasoning remain hard for current models.
The shared library enables consistent, reproducible addition of new tasks without redesigning the physics layer.
Results establish a baseline for future work on language-to-structure pipelines that must satisfy engineering standards.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The benchmark format could be extended to include visual feedback loops, allowing models to revise plans after simulated failures.
Strong performance here would suggest LLMs are ready for hybrid systems that combine language planning with physics simulators in robotics.
The approach may transfer to related domains such as architectural design or disaster-response structure assembly.
A natural next measurement is whether models improve when given access to the same geometric library during inference.

Load-bearing premise

The proposed task design strategy and 3D Spatial Geometric Computation Library accurately capture the physical constraints and reasoning demands of real-world engineering construction automation.

What would settle it

Compare LLM-guided construction outcomes in BuildArena against the same models directing physical robots or real construction equipment and check whether success rates and failure modes match.

Figures

Figures reproduced from arXiv: 2510.16559 by Chenglei Yu, Long Wei, Tailin Wu, Tianrun Gao, Tian Xia, Wenhao Deng, Xiaowei Qian.

**Figure 1.** Figure 1: Examples of BuildArena’s construction results by LLMs, covering three tasks: Lift (left subfigure), Transport (upper right), and Support (lower right). ABSTRACT Engineering construction automation aims to transform natural language specifications into physically viable structures, requiring complex integrated reasoning under strict physical constraints. While modern LLMs possess broad knowledge and strong… view at source ↗

**Figure 2.** Figure 2: Illustration of our BuildArena framework. It contains three parts: (1) Task definition; (2) LLM-based Construction; (3) Simulation-based Evaluation. The arrows represent our pipeline. Components in dashed boxes, i.e., task type, LLM agentic workflow, and simulator, could be customized by users. Details of the construction procedure is shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Difficulty profiles of the three task Transport, Support, and Lift across six engineering dimensions: Quantification, Redundancy, Scale, Modularity, Precision, and Ambiguity. Each radar chart illustrates how difficulty escalates from Lv.1 (blue) to Lv.2 (purple) and Lv.3 (red). Lift requires constructing a rocket. At Lv.1, LLMs are explicitly required to build a single rocket engine without instruction on… view at source ↗

**Figure 4.** Figure 4: Details of the construction procedure in Figure 2. Our designed workflow (bottom row) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

**Figure 5.** Figure 5: Example of the construction process. The rocket is constructed by Grok-4 for the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Distributions of failure reasons averaged over different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 7.** Figure 7: Performance of different LLMs against six dimensions of task difficulty: Quantification (Q), Robustness (R), Magnitude (M), Compositionality (C), Precision (P), Ambiguity (A). The performance of different LLMs across six task difficulty dimensions is presented in Figure 7. It calculates the weighted score of each LLM across all the difficulty dimensions based on its score ranking in each task, followed b… view at source ↗

**Figure 8.** Figure 8: Trade-off between performance and cost. Longer output does not imply better results. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗

**Figure 9.** Figure 9: More examples of construction results of [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

**Figure 10.** Figure 10: Building procedure examples across 9 tasks. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

read the original abstract

Engineering construction automation aims to transform natural language specifications into physically viable structures, requiring complex integrated reasoning under strict physical constraints. While modern LLMs possess broad knowledge and strong reasoning capabilities that make them promising candidates for this domain, their construction competencies remain largely unevaluated. To address this gap, we introduce BuildArena, the first physics-aligned interactive benchmark designed for language-driven engineering construction. It takes a first step towards engineering automation using LLMs. Technically, it contributes to the community in two aspects:(1) an extendable task design strategy spanning static and dynamic mechanics across multiple difficulty tiers; (2) a 3D Spatial Geometric Computation Library for supporting construction based on language instructions. On nine frontier LLMs, BuildArena comprehensively evaluates their capabilities for language-driven and physics-grounded construction automation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

BuildArena adds a benchmark and geometric library for language-driven 3D construction tasks, but the physics alignment rests on thin evidence and may not reach full dynamics.

read the letter

The main thing to know is that this paper introduces BuildArena as the first benchmark pairing natural language instructions with interactive 3D construction tasks that claim physics alignment, backed by an extendable task strategy across static and dynamic mechanics plus a 3D Spatial Geometric Computation Library. They run it on nine frontier LLMs to show where current models stand on turning specs into viable structures. That setup is genuinely new for the engineering automation angle and gives a concrete way to move beyond abstract reasoning tests. The library and tiered task design are the parts that could actually see use by others working on grounded LLM applications. The evaluation provides an initial data point on model performance in this domain, which is a reasonable starting contribution. The soft spot is the physics grounding itself. The stress-test concern holds up on the available description: if the library mainly handles geometric checks like intersections, volumes, and poses without a dynamics solver for forces, contact, friction, or stability, then the dynamic-mechanics tasks test spatial arrangement more than real physical viability. The abstract and task strategy emphasize physics alignment, yet no details appear on validation against ground-truth simulations or material failure checks. That leaves the central claim under-supported. Minor issues like missing calibration details on task difficulty could be fixed in revision, but the physics implementation is the load-bearing part that needs clearer evidence. This paper is for researchers building or using benchmarks in robotics, embodied AI, or construction automation. Readers who want new evaluation frameworks for physically grounded reasoning will get value from the task design and library even if they question the results. It deserves a serious referee because the benchmark idea is fresh and the domain gap it targets is real, though revisions should focus on showing how the library actually enforces dynamic constraints rather than just kinematics.

Referee Report

2 major / 2 minor

Summary. The paper introduces BuildArena as the first physics-aligned interactive benchmark for evaluating LLMs on language-driven engineering construction tasks. It proposes an extendable task design strategy spanning static and dynamic mechanics across difficulty tiers and contributes a 3D Spatial Geometric Computation Library to enable construction from natural language instructions. The work evaluates nine frontier LLMs on these tasks to assess their capabilities for physics-grounded construction automation.

Significance. If the physics alignment of the tasks and library is rigorously validated against ground-truth simulations and the benchmark tasks prove to capture real engineering constraints, BuildArena could serve as a useful standardized testbed for measuring progress in LLM-based construction automation, particularly by exposing limitations in spatial and physical reasoning that current models exhibit.

major comments (2)

[Abstract and §3] Abstract and §3 (Task Design and Library): The central claim that BuildArena provides 'physics-aligned' and 'physics-grounded' evaluation for dynamic mechanics tasks is not supported by evidence of integration with a dynamics engine. The 3D Spatial Geometric Computation Library appears limited to geometric intersection, volume, and pose checks; without explicit handling of forces, gravity, contact dynamics, friction, or stability analysis, dynamic-tier tasks risk evaluating only kinematic/spatial reasoning rather than the intended physical constraints.
[Evaluation] Evaluation section: No quantitative results, validation metrics for physics alignment, or task difficulty calibration details are provided in the abstract or visible summary. This leaves the empirical support for claims about LLM performance on the benchmark without visible grounding, undermining the ability to assess whether the nine-LLM evaluation demonstrates meaningful physics-grounded capabilities.

minor comments (2)

[Abstract] The abstract states the benchmark 'takes a first step' but does not clarify how the extendable task design strategy ensures coverage of material failure or structural integrity beyond geometry.
[§3] Notation for difficulty tiers and mechanics categories could be more explicitly defined with examples to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, providing clarifications on the physics alignment approach and committing to revisions that improve the visibility of our evaluation results and the precise scope of our claims.

read point-by-point responses

Referee: [Abstract and §3] Abstract and §3 (Task Design and Library): The central claim that BuildArena provides 'physics-aligned' and 'physics-grounded' evaluation for dynamic mechanics tasks is not supported by evidence of integration with a dynamics engine. The 3D Spatial Geometric Computation Library appears limited to geometric intersection, volume, and pose checks; without explicit handling of forces, gravity, contact dynamics, friction, or stability analysis, dynamic-tier tasks risk evaluating only kinematic/spatial reasoning rather than the intended physical constraints.

Authors: We appreciate the referee's careful reading. The 3D Spatial Geometric Computation Library is designed to perform geometric operations (intersection, volume, and pose checks) that serve as practical proxies for enforcing physical constraints in construction tasks. The 'physics-aligned' designation stems from the task design strategy, which structures dynamic mechanics problems around outcomes that must satisfy stability and feasibility conditions approximated through these geometric validations. We acknowledge that this implementation does not include a full dynamics engine with explicit force, gravity, friction, or contact modeling. To strengthen the manuscript, we will revise the abstract and §3 to explicitly describe the current geometric-proxy approach, state its limitations relative to full physical simulation, and note that future extensions could incorporate dynamics engines. This revision will ensure the claims accurately reflect the technical scope without overstatement. revision: yes
Referee: [Evaluation] Evaluation section: No quantitative results, validation metrics for physics alignment, or task difficulty calibration details are provided in the abstract or visible summary. This leaves the empirical support for claims about LLM performance on the benchmark without visible grounding, undermining the ability to assess whether the nine-LLM evaluation demonstrates meaningful physics-grounded capabilities.

Authors: The full evaluation section reports quantitative results, including success rates and failure modes for the nine evaluated LLMs across static and dynamic task tiers. Physics alignment is validated by comparing the geometric library's outputs against the physical viability criteria defined in each task. Task difficulty calibration is achieved through the progressive design in §3, where tiers increase in spatial complexity and dynamic requirements. To address the concern about visibility, we will revise the abstract to include key quantitative highlights (e.g., overall performance ranges) and add a short summary of the validation and calibration procedures. These changes will make the empirical grounding more immediately accessible while preserving the detailed analysis in the main text. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark and library introduced as new external artifacts

full rationale

The paper introduces BuildArena as a new interactive benchmark along with an extendable task design strategy and 3D Spatial Geometric Computation Library. No derivations, first-principles results, fitted parameters, or predictions are claimed that could reduce to the paper's own inputs by construction. The central contributions are the creation and description of these artifacts for evaluating LLMs on language-driven construction tasks spanning static and dynamic mechanics. Evaluations are performed on external frontier LLMs rather than self-referential data. No self-citation chains, uniqueness theorems, or ansatzes are invoked to justify load-bearing steps. The work is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper introduces a benchmark and library without detailing free parameters, axioms, or invented entities in the abstract; the central contribution rests on the assumption that the described task design and library adequately model physical construction.

pith-pipeline@v0.9.0 · 5676 in / 1074 out tokens · 32445 ms · 2026-05-21T20:10:52.598680+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/AlexanderDuality.lean alexander_duality_circle_linking unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

a 3D Spatial Geometric Computation Library for supporting construction based on language instructions... physical constraint checks: ... collision avoidance

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization
cs.AI 2026-04 unverdicted novelty 6.0

Frontier-Eng is a new benchmark for generative optimization in engineering where agents iteratively improve designs under fixed interaction budgets using executable verifiers, with top models like GPT 5.4 showing limi...

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 1 Pith paper

[1]

Envision an overall structure that can achieve the goal

work page
[2]

If necessary, break down this structure into non-redundant and reusable basic sub-structures or components, each sub-structure should be constructed independently, and the final structure will be assembled by attaching or connecting the sub-structures together

work page
[3]

For each sub-structure, determine which building blocks will be used and how they will be arranged

work page
[4]

Consider how these sub-structures will be assembled to form the complete structure

work page
[5]

Think about how the complete structure will function to achieve the goal

work page
[6]

Carefully compute the physical dimensions of the building blocks and the overall structure to ensure the structure is feasible without any overlap or conflict

work page
[7]

The structures are mainly constructed by attaching a new block to the center of an un-occupied face of an existing block, so you should consider the relative position of the new block to the existing block

work page
[8]

Positions may be micro- adjusted in later stages to resolve conflicts based on actual build execution

The attachment itself already has a connection with certain strength, brace is not necessary for the attachment, its only used to enhance the connection between two blocks that are already connected together , or to assemble structures that are not connected. Your final output should be structured in the following format: <building_plan> <overall_structur...

work page
[9]

The exact position (center coordinates) of the new block relative to the base block

work page
[10]

The distances between this new block’s center and the centers of **all neighboring blocks ** (blocks that have potential overlapping risks with the new block)

work page
[11]

- Any overlap or improper attachment must be flagged explicitly

Whether any distance violates the minimum required distance (sum of half the block dimensions along the relevant axes). - Any overlap or improper attachment must be flagged explicitly. FUNCTIONAL VALIDATION: - Check each point in detail, reasoning logically before proceeding to the next. Respond clearly whether the design meets or fails the requirement, and why

work page
[12]

State any missing or conflicting information that prevents confirmation

Verify that the described structure allows the specified motion (e.g., rotation, translation). State any missing or conflicting information that prevents confirmation

work page
[13]

For all functional components (e.g., wheels, cannon, etc.), carefully calculate their parameters (e.g., direction of motion, direction of shooting, etc.) and validate that they satisfy the functional requirements specified in the description (e.g., axis alignment, motion direction)

work page
[14]

UpArrow",

Verify moving components have appropriate mounting and alignment. Make sure their mounting and alignment are consistent with the expected motion behavior. REVIEW PROCESS: - First, **systematically check structural integrity and collision-free placement one block at a time ** as outlined above. - Then, validate functional implementation. - Finally, assess ...

work page

[1] [1]

Envision an overall structure that can achieve the goal

work page

[2] [2]

If necessary, break down this structure into non-redundant and reusable basic sub-structures or components, each sub-structure should be constructed independently, and the final structure will be assembled by attaching or connecting the sub-structures together

work page

[3] [3]

For each sub-structure, determine which building blocks will be used and how they will be arranged

work page

[4] [4]

Consider how these sub-structures will be assembled to form the complete structure

work page

[5] [5]

Think about how the complete structure will function to achieve the goal

work page

[6] [6]

Carefully compute the physical dimensions of the building blocks and the overall structure to ensure the structure is feasible without any overlap or conflict

work page

[7] [7]

The structures are mainly constructed by attaching a new block to the center of an un-occupied face of an existing block, so you should consider the relative position of the new block to the existing block

work page

[8] [8]

Positions may be micro- adjusted in later stages to resolve conflicts based on actual build execution

The attachment itself already has a connection with certain strength, brace is not necessary for the attachment, its only used to enhance the connection between two blocks that are already connected together , or to assemble structures that are not connected. Your final output should be structured in the following format: <building_plan> <overall_structur...

work page

[9] [9]

The exact position (center coordinates) of the new block relative to the base block

work page

[10] [10]

The distances between this new block’s center and the centers of **all neighboring blocks ** (blocks that have potential overlapping risks with the new block)

work page

[11] [11]

- Any overlap or improper attachment must be flagged explicitly

Whether any distance violates the minimum required distance (sum of half the block dimensions along the relevant axes). - Any overlap or improper attachment must be flagged explicitly. FUNCTIONAL VALIDATION: - Check each point in detail, reasoning logically before proceeding to the next. Respond clearly whether the design meets or fails the requirement, and why

work page

[12] [12]

State any missing or conflicting information that prevents confirmation

Verify that the described structure allows the specified motion (e.g., rotation, translation). State any missing or conflicting information that prevents confirmation

work page

[13] [13]

For all functional components (e.g., wheels, cannon, etc.), carefully calculate their parameters (e.g., direction of motion, direction of shooting, etc.) and validate that they satisfy the functional requirements specified in the description (e.g., axis alignment, motion direction)

work page

[14] [14]

UpArrow",

Verify moving components have appropriate mounting and alignment. Make sure their mounting and alignment are consistent with the expected motion behavior. REVIEW PROCESS: - First, **systematically check structural integrity and collision-free placement one block at a time ** as outlined above. - Then, validate functional implementation. - Finally, assess ...

work page