Assessment of ChatGPT for Engineering Statics Analysis

Benjamin Hope; Derek Warner; Jayden Bracey; Sahar Choukir

REVIEW 3 major objections 5 minor 24 references

Assessment of ChatGPT for Engineering Statics Analysis

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A prompt-engineered Custom GPT scored above the average first-year engineering student on text-based statics exams, while still failing on nuanced and image-based problems.

desk verdict Useful preliminary prompt-engineering benchmark for ChatGPT in engineering statics, but the 'surpasses student average' headline is a single stochastic draw, not a measurement. read the letter →

arxiv 2502.00562 v1 pith:LEXW6NID submitted 2025-02-01 cs.CE

Benjamin Hope , Jayden Bracey , Sahar Choukir , Derek Warner This is my paper · ORCID

classification cs.CE

keywords ChatGPTlargelanguagemodelsengineeringstaticspromptchain-of-thoughtpromptingtrussanalysisbeameducation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether ChatGPT can be trusted for basic engineering statics analysis and compares it with first-year engineering students. It finds that a Custom GPT with refined prompts embedded in its instructions scores 82% on one statics exam and 86% on a second, exceeding the respective student averages of 75% and 81%. The gain comes mainly from combining chain-of-thought reasoning with explicit instructions about force directions, angles, and sign conventions. The same models continue to make persistent errors on nuanced problems, most notably misidentifying tension and compression in truss members, and image-based prompts sharply degrade performance. The authors interpret this as evidence that LLMs rely on pattern recognition from training data rather than systematic application of engineering principles.

What carries the argument

The machinery is the Custom GPT, a ChatGPT instance whose system instructions embed the most effective prompt elements found in the study: chain-of-thought reasoning, explicit reminders about sign conventions and force directions, and the rule to use Python for all calculations to avoid arithmetic errors. These embedded instructions carry the argument because the headline exam scores of 82% and 86% come from this configured model rather than from stock ChatGPT. The study also tests seven prompt styles, with Style 5, which combines chain-of-thought reasoning with the sign-convention note, yielding the highest scores.

What would settle it

Run the Custom GPT on Exam 2 dozens of times at temperature 0.7 and plot the score distribution; if the spread spans the student average, or if a class of students scores markedly higher on Exam 2 than on Exam 1, the claim that the model outperforms students is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that a domain-tuned Custom GPT, built by embedding the study's best-performing prompts into ChatGPT's instructions, can score at or above the average first-year student on a standard statics exam when the questions are given as text. On Exam 1 the Custom GPT scored 82% against a 705-student average of 75%; on Exam 2 it scored 86% against a 7-student average of 81%. The improvement came from combining chain-of-thought reasoning with explicit reminders about force directions, angles, and sign conventions, while image-based prompts consistently produced worse results. The authors conclude that ChatGPT solves these problems mainly through pattern recognition rather than deductive equilibrium analysis, which explains both the exam-level performance and the recurring errors in tension versus compression classification and inclined-force decomposition.

Load-bearing premise

The argument that the Custom GPT beats the average student assumes that Exam 1 and Exam 2 are equally difficult, even though the second exam was taken by only seven students and the equivalence was never tested.

Editorial extensions

If this is right

Chain-of-thought prompting combined with explicit sign-convention instructions produces the highest exam scores among the tested prompt styles.
A Custom GPT with these instructions embedded scored 82% on Exam 1 and 86% on Exam 2, exceeding the student averages of 75% and 81%.
Image-based prompts produced the poorest results, indicating that ChatGPT-4o's current multimodal reading is not reliable for statics diagrams.
Even the optimized model continued to misidentify tension versus compression in truss members and mishandle inclined forces, so fully correct solutions on nuanced problems remained rare.
The authors conclude that LLMs solve statics problems primarily through pattern recognition rather than deductive application of equilibrium principles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The headline comparison rests on single runs at temperature 0.7; repeated sampling would likely show score variance, so the 'beats the student average' result is less stable than it appears.
A direct extension would be to calibrate Exam 1 and Exam 2 by having the same students take both; the paper explicitly assumes rather than tests their equivalence.
For educational use, the results imply that LLM outputs should be treated as drafts to be checked, because confident errors in tension and compression classification can pass as correct to a novice.
Future work could test whether the same prompt-engineering gains transfer to other mechanics topics or to problems requiring free-body diagrams extracted from images.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a worthwhile preliminary benchmark, but the headline claim that the Custom GPT 'surpasses the student average' is not statistically grounded. The 82% and 86% scores are single draws from a stochastic model at temperature 0.7, and the paper itself tells you response variability is substantial.

What's actually new: applying the established LLM-exam evaluation design to engineering statics, with seven prompt styles and a custom GPT. The beam and truss case studies are concrete, and the authors clearly document failure modes — tension/compression misclassification, hallucinated forces, poor image-based performance. They are not overselling; they explicitly say the tool is not a replacement for human expertise.

Soft spots, in order of importance. First, single-run results. Section 2.2 says temperature introduces randomness, yet Section 4 reports each exam as a single point with no error bars. The 82 vs 75 and 86 vs 81 comparisons are anecdotal; a few different draws would likely move the point estimate across the student mean. Second, the Custom GPT's Exam 2 score is confounded: the model was given Exam 1 and its marking rubric, and Exam 2 is described as containing similar problems with minor variations. That is few-shot leakage, not a clean test. Third, the paper assumes Exam 1 and Exam 2 are comparable; the assumption is untested but doesn't affect the primary 82 vs 75 comparison. Fourth, Figure 1's 'R^2' is used loosely (it's a correlation coefficient, not a coefficient of determination). Minor.

The central argument is not internally broken; the evidence is just thin. This is a legitimate pilot study that a good revision could turn into a solid contribution. Add repeated sampling, error bars, a held-out exam, and raw data availability.

Recommendation: send it to peer review. A serious referee will push the authors to fix the statistical reporting and the leakage issue. I'd cite the paper for its prompt-engineering comparison, but not for the headline numbers.

Referee Report

3 major / 5 minor

Summary. The paper compares ChatGPT-4o, ChatGPT-o1-preview, and a custom prompt-engineered GPT on engineering statics problems, ranging from F=ma calculations to beam and truss analyses, and on two first-year statics exams. The authors report that the Custom GPT scored 82% on Exam 1 and 86% on Exam 2, surpassing student averages of 75% and 81%, respectively. The central claims are that prompt engineering substantially improves LLM performance on routine statics, but that image-based and nuanced open-ended problems remain unreliable. The paper also documents specific failure modes such as tension/compression misclassification and difficulty decomposing inclined forces.

Significance. If the headline comparisons were statistically grounded, the paper would provide a useful benchmark of current LLM ability in a core engineering course, with practical implications for assessment design and AI-assisted tutoring. The detailed qualitative documentation of failure modes is a genuine strength, as is the transparent disclosure of the Custom GPT's system-prompt accessibility and the use of a common grading rubric. However, the main quantitative claim is currently supported by single stochastic runs without error bars, and the evaluation protocol leaks information from Exam 1 into the Exam 2 assessment. These shortcomings are fixable but materially affect the current conclusions.

major comments (3)

[Section 2.2 and Section 4.2] The headline comparisons (Custom GPT 82% vs 75% on Exam 1; 86% vs 81% on Exam 2) rest on single stochastic runs at temperature 0.7, yet Section 2.2 explicitly states that response variability is substantial even for identical prompts. The paper reports these scores as deterministic point values with no error bars, no repeated sampling, and no significance testing. A few favorable or unfavorable draws could place the expected score on either side of the student mean. This is load-bearing for the abstract's claim that the Custom GPT 'surpasses the 75% student average.' The authors should either report means and confidence intervals from multiple runs (e.g., 10–20 repetitions) or explicitly downgrade the claim to 'a single run achieved 82%,' which would remove the inferential force.
[Section 4.1 and Section 4.2] The evaluation of the Custom GPT on Exam 2 is contaminated by information leakage. Section 4.1 states that 'Exam 1, along with its marking rubric, was also provided to the Custom GPT model for use in solving Exam 2,' and Exam 2 is described as containing similar problems with minor variations. This gives the Custom GPT a worked template for its Exam 2 attempt, so the 86% score is not a clean test of prompt-engineering alone. The comparison to the 81% student average is therefore not a like-for-like assessment. Please either withhold all worked examples and rubrics from the model for a held-out exam, or clearly re-frame the result as conditional on having access to a near-identical prior exam and rubric.
[Section 4.2] The comparison of the Custom GPT's 86% on Exam 2 against the 81% student average for Exam 2 depends on the untested assumption that 'the difficulty levels of the two versions were comparable.' Only seven students took Exam 2, and no evidence is given that the two exams are equated in difficulty, marking leniency, or problem distribution. The paper itself acknowledges this assumption. Please provide sensitivity analysis: for example, state how much easier Exam 2 would need to be than Exam 1 for the 86% result to cease being meaningful, and discuss the high uncertainty implied by n=7. In the current form, the 86%-vs-81% comparison should be presented as provisional at best.

minor comments (5)

[Section 1, paragraph 3] The phrase 'This study seeks to addresses this gap' should read 'This study seeks to address this gap.'
[Section 3.1, Figure 1 caption and text] The text reports 'a correlation coefficient (R2) of 0.79'; this conflates R and R². Please clarify whether this is Pearson's r or the coefficient of determination, and report both if available. Also, the 100-iteration results lack raw data or per-trial error information, making the reported accuracy difficult to assess.
[Section 3.2.2 and Section 5] Typos such as 'wether' (in Section 3.2.2) and 'a a first-year' (in Section 5) should be corrected. The repeated 'aster' should be 'after'.
[Section 4.1, prompt style list] The list of prompt styles is clear, but it would help to specify how many questions constituted each exam and how partial credit was awarded. This would improve reproducibility of the grading procedure.
[Section 4.2] The statement that the Custom GPT scored 82% on Exam 1 and 'within the top 36% of student results' is slightly ambiguous; please clarify whether this means the top 36% (i.e., 64th percentile or higher) or the 36th percentile.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity in the derivation sense; the study is an empirical benchmark with no fitted predictions or self-citation chains.

full rationale

This paper makes no mathematical derivation whose outputs are definitionally equal to its inputs. The central claim is empirical: a Custom GPT scored 82% on Exam 1 and 86% on Exam 2, compared with student averages of 75% and 81%. These scores are benchmarked against external exams adapted from standard statics textbooks, and the paper states the exam was 'graded consistently for both students and ChatGPT models using a common rubric.' No free parameter is fitted to the reported exam outcomes and then renamed a prediction. The Custom GPT was developed by consolidating prompt styles identified during earlier testing, which is iterative prompt engineering rather than fitting a model to the target exam data. The statement that 'Exam 1, along with its marking rubric, was also provided to the Custom GPT model for use in solving Exam 2' is a potential information-leakage confound for the Exam 2 comparison, but it does not make the claimed score equivalent to an input by construction. The unverified assumption that Exam 1 and Exam 2 difficulty levels are comparable, and the use of single stochastic runs at temperature 0.7 without repeated sampling, are statistical-validity concerns rather than circularity. The paper contains no load-bearing self-citations and no imported uniqueness theorem. The circularity burden is therefore low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and uses no fitted mathematical parameters. It does, however, rest on three unproven domain assumptions: exam comparability, single-run representativeness, and rubric validity. The Custom GPT itself is a proprietary artifact, not a postulated physical entity, so no invented entities are listed.

assumptions (3)

domain assumption Exam 1 and Exam 2 are comparable in difficulty.
Section 4.1: 'Due to the small sample size for Exam 2, it was assumed that the difficulty levels of the two versions were comparable to allow for meaningful comparisons.' The headline Custom GPT versus student comparison relies on this.
domain assumption Model outputs at temperature 0.7 with a single run are representative of model performance.
Section 2.2 acknowledges response variability from non-zero temperature, yet the reported exam scores appear to come from single runs without repeated sampling or error bars.
domain assumption The authors' rubric captures meaningful engineering correctness.
Section 4.1 states both students and models were graded with a common rubric, but the rubric is not fully shown and inter-rater reliability is not reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Assessment of ChatGPT for Engineering Statics Analysis." pith.science (2026). https://pith.science/paper/LEXW6NID

@misc{pith2026250200562,
  author       = {Pith},
  title        = {Pith review of: Assessment of ChatGPT for Engineering Statics Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEXW6NID}},
  note         = {Machine review of arXiv:2502.00562}
}

read the original abstract

Large language models (LLMs) such as OpenAI's ChatGPT hold potential for automating engineering analysis, yet their reliability in solving multi-step statics problems remains uncertain. This study evaluates the performance of ChatGPT-4o and ChatGPT-o1-preview on foundational statics tasks, from simple calculations of Newton's second law of motion to beam and truss analyses and compares their results to first-year engineering students on a typical statics exam. To enhance accuracy, we developed a Custom GPT, embedding refined prompts directly into its instructions. This optimized model achieved an 82% score, surpassing the 75% student average, demonstrating the impact of tailored guidance. Despite these improvements, LLMs continued to exhibit errors in nuanced or open-ended problems, such as misidentifying tension and compression in truss members. These findings highlight both the promise and current limitations of AI in structural analysis, emphasizing the need for improved reasoning, multimodal capabilities, and targeted training data for future AI-driven automation in civil and mechanical engineering.

Figures

Figures reproduced from arXiv: 2502.00562 by the authors.

**Figure 1.** Accuracy of ChatGPT in calculating F = m · a across 100 iterations. The mass (m) and acceleration (a) values were randomly generated by ChatGPT for each calculation. The y-axis represents the force (F) values calculated by ChatGPT, and the x-axis represents the correct force values calculated using Python. The blue line represents perfect accuracy. instruction to use Python for all computations was maintained throug… view at source ↗

**Figure 2.** Beam problem presented to ChatGPT ([21]). [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Truss problem presented to ChatGPT [22]. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Incorrect diagram output produced by ChatGPT-4o from image-based prompt. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

**Figure 5.** Figure 5: 2024 First-Year Engineering Statics Exam referred to Exam 2. Problems adapted [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

**Figure 6.** Figure 6: Results comparing different prompt styles used to evaluate the GPT-4o and [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

**Figure 7.** Figure 7: Results comparing the Custom GPT against first-year engineering statics exams. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

[1]

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837

2022
[2]

Joshi, R

I. Joshi, R. Budhiraja, P. D. Tanna, L. Jain, M. Deshpande, A. Sri- vastava, S. Rallapalli, H. D. Akolekar, J. Sesh Challa, D. Kumar, ” with great power comes great responsibility!”: Student and instructor perspectives on the influence of llms on undergraduate engineering edu- cation, arXiv e-prints (2023) arXiv–2309

work page 2023
[3]

A Preliminary Exploration of the Disruption of a Generative AI Systems: Faculty/Staff and Student Perceptions of ChatGPT and its Capability of Completing Undergraduate Engineering Coursework

L. White, T. Balart, S. Amani, K. J. Shryock, K. L. Watson, A preliminary exploration of the disruption of a generative ai systems: Faculty/staff and student perceptions of chatgpt and its capability of completing undergraduate engineering coursework, arXiv preprint arXiv:2403.01538 (2024)

work page Pith review arXiv 2024
[4]

M. E. Frenkel, H. Emara, Chatgpt-3.5 and-4.0 and mechanical engi- neering: Examining performance on the fe mechanical engineering and undergraduate exams, Computer Applications in Engineering Education (2024) e22781

work page 2024
[5]

S. J. Uddin, A. Albert, M. Tamanna, A. Ovid, A. Alsharef, Chatgpt as an educational resource for civil engineering students, Computer Appli- cations in Engineering Education (2024) e22747. 24

work page 2024
[6]

B. Ni, M. J. Buehler, Mechagents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge, Extreme Mechanics Letters 67 (2024) 102131

2024
[7]

Quibeldey-Cirkel, Disrupting stem education: the game-changing im- pact of chatgpt and chatgpt codex, in: Edulearn23 Proceedings, IATED, 2023, pp

K. Quibeldey-Cirkel, Disrupting stem education: the game-changing im- pact of chatgpt and chatgpt codex, in: Edulearn23 Proceedings, IATED, 2023, pp. 220–228

work page 2023
[8]

Morsy, A

M. Morsy, A. Farraj, D. Reavis, On the challenges and opportunities of using chatgpt in academia, in: 2023 IEEE Frontiers in Education Conference (FIE), IEEE, 2023, pp. 01–06

work page 2023

Show all 24 references

[9]

Zeeshan, T

K. Zeeshan, T. H¨ am¨ al¨ ainen, P. Neittaanm¨ aki, Chatgpt for stem edu- cation: A working framework, International Journal of Learning and Teaching 10 (4) (2024)

2024
[10]

S. Saad, M. Haris, S. Ammad, K. Rasheed, Ai-assisted building design, in: AI in Material Science, CRC Press, 2024, pp. 143–168

2024
[11]

V. L. Bommineni, S. Bhagwagar, D. Balcarcel, V. Bommineni, C. Davaz- itkos, D. Boyer, Performance of chatgpt on the mcat: the road to per- sonalized and equitable premedical learning, MedRxiv (2023) 2023–03

2023
[12]

Hickman, P

L. Hickman, P. D. Dunlop, J. L. Wolf, The performance of large language models on quantitative and verbal ability tests: Initial evidence and implications for unproctored high-stakes testing, International Journal of Selection and Assessment (2024)

2024
[13]

E. J. L. Tan, K. A. L. Ramos, M. E. K. B. Nazario, S. V. D. Lim, S. B. Chu, Ai to the test: Measuring chatgpt’s objective accuracy in the sats in comparison to human performance, in: 2024 IEEE 48th An- nual Computers, Software, and Applications Conference (COMPSAC), IEEE, 2024...

2024
[14]

Kipp, From gpt-3.5 to gpt-4

M. Kipp, From gpt-3.5 to gpt-4. o: A leap in ai’s medical exam perfor- mance, Information 15 (9) (2024) 543

2024
[15]

Newton, M

P. Newton, M. Xiromeriti, Chatgpt performance on multiple choice ques- tion examinations in higher education. a pragmatic scoping review, As- sessment & Evaluation in Higher Education 49 (6) (2024) 781–798. 25

2024
[16]

OpenAI, Gpt-4 technical report

R. OpenAI, Gpt-4 technical report. arxiv 2303.08774, View in Article 2 (5) (2023)

2023 arXiv
[17]

Latif, Y

E. Latif, Y. Zhou, S. Guo, Y. Gao, L. Shi, M. Nayaaba, G. Lee, L. Zhang, A. Bewersdorff, L. Fang, et al., A systematic assessment of openai o1-preview for higher order thinking in education, arXiv preprint arXiv:2410.21287 (2024)

2024 arXiv
[18]

H. Dang, L. Mecke, F. Lehmann, S. Goller, D. Buschek, How to prompt? opportunities and challenges of zero-and few-shot learning for human-ai interaction in creative applications of generative models, arXiv preprint arXiv:2209.01390 (2022)

2022 arXiv
[19]

Schulhoff, M

S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y. Li, A. Gupta, H. Han, S. Schulhoff, et al., The prompt report: A systematic survey of prompting techniques, arXiv preprint arXiv:2406.06608 (2024)

2024 arXiv
[20]

T. B. Brown, Language models are few-shot learners, arXiv preprint arXiv:2005.14165 (2020)

2020 arXiv
[21]

R. C. Hibbeler, Statics and Mechanics of Materials in SI Units, Pearson Higher Ed, 2018

2018
[22]

C. M. Fleischmann, Engr102 statics lecture 10, Lecture delivered at University of Canterbury, lecture (2019)

2019
[23]

F. P. Beer, E. Johnston Jr, M. Russell, F. David, E. R. Eisenberg, Vector mechanics for engineers: Statics (si units), McGraw Hill Higher Education, 9th Revised edition (October 2010), Cap ´ ıtulo 3 (2019) (2019) 75

2019
[24]

F. Beer, E. Johnston, J. DeWolf, Mechanics of materials, 5th si edition, Stress 1 (10) (1999) 1–12. 26

1999

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837

2022

[2] [2]

Joshi, R

I. Joshi, R. Budhiraja, P. D. Tanna, L. Jain, M. Deshpande, A. Sri- vastava, S. Rallapalli, H. D. Akolekar, J. Sesh Challa, D. Kumar, ” with great power comes great responsibility!”: Student and instructor perspectives on the influence of llms on undergraduate engineering edu- cation, arXiv e-prints (2023) arXiv–2309

work page 2023

[3] [3]

A Preliminary Exploration of the Disruption of a Generative AI Systems: Faculty/Staff and Student Perceptions of ChatGPT and its Capability of Completing Undergraduate Engineering Coursework

L. White, T. Balart, S. Amani, K. J. Shryock, K. L. Watson, A preliminary exploration of the disruption of a generative ai systems: Faculty/staff and student perceptions of chatgpt and its capability of completing undergraduate engineering coursework, arXiv preprint arXiv:2403.01538 (2024)

work page Pith review arXiv 2024

[4] [4]

M. E. Frenkel, H. Emara, Chatgpt-3.5 and-4.0 and mechanical engi- neering: Examining performance on the fe mechanical engineering and undergraduate exams, Computer Applications in Engineering Education (2024) e22781

work page 2024

[5] [5]

S. J. Uddin, A. Albert, M. Tamanna, A. Ovid, A. Alsharef, Chatgpt as an educational resource for civil engineering students, Computer Appli- cations in Engineering Education (2024) e22747. 24

work page 2024

[6] [6]

B. Ni, M. J. Buehler, Mechagents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge, Extreme Mechanics Letters 67 (2024) 102131

2024

[7] [7]

Quibeldey-Cirkel, Disrupting stem education: the game-changing im- pact of chatgpt and chatgpt codex, in: Edulearn23 Proceedings, IATED, 2023, pp

K. Quibeldey-Cirkel, Disrupting stem education: the game-changing im- pact of chatgpt and chatgpt codex, in: Edulearn23 Proceedings, IATED, 2023, pp. 220–228

work page 2023

[8] [8]

Morsy, A

M. Morsy, A. Farraj, D. Reavis, On the challenges and opportunities of using chatgpt in academia, in: 2023 IEEE Frontiers in Education Conference (FIE), IEEE, 2023, pp. 01–06

work page 2023