Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering

Christophe Bajan; Guillaume Lambard

REVIEW 3 major objections 4 minor 16 references

Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Closed-source LLMs answer materials-science exam questions with about 84 percent accuracy, while the best open-source model reaches only 56 percent.

desk verdict A useful and honest benchmark sweep of 15 LLMs on MaScQA; the headline ranking is credible, but the absolute numbers for weak open-source models are shakier than the tables suggest because the automated judge was validated on only 4 of the 15 models. read the letter →

arxiv 2501.04277 v1 pith:VSN6R735 submitted 2025-01-08 physics.comp-ph cond-mat.mtrl-sci

Christophe Bajan , Guillaume Lambard This is my paper · ORCID

classification physics.comp-phcond-mat.mtrl-sci

keywords LLMevaluationmaterialsscienceMaScQAbenchmarkquestionansweringnumericalreasoningopen-sourcelanguagemodelsLLM-as-a-judgemetallurgicalengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to measure how well existing large language models know materials science before any domain-specific training. Using the MaScQA benchmark, a set of 644 exam questions from the Graduate Aptitude Test in Engineering covering four question formats and fourteen sub-fields, it evaluates fifteen models and reports that the closed-source Claude-3.5-Sonnet and GPT-4o answer about 84 percent correctly. The best open-source model, Llama3-70b, reaches about 56 percent, and the smaller Phi3-14b about 43 percent. These numbers matter because the authors need a baseline against which to measure a planned fine-tuned open-source model and retrieval-augmented generation system for materials science.

What carries the argument

The central machinery is the MaScQA benchmark itself plus the scoring pipeline built around it. MaScQA contributes 644 questions divided into four types — MATCH (match the following), MCQ (multiple choice), MCQN (multiple choice numerical), and NUM (open numerical) — which together test pattern matching, conceptual knowledge, and arithmetic. To score roughly 37,000 model answers without manual grading, the authors use an LLM-as-a-judge setup in which GPT-4o classifies each answer against the gold answer, using a revised prompt that instructs the judge to check both the selected option and any accompanying reasoning. They also fix the generation temperature to 0, submit each question three times, and compare every model against a random baseline derived from the binomial distribution with success probability $p = 0.25$ for the multiple-choice formats and zero for NUM. The benchmark defines what counts as success, the judge converts responses into accuracy numbers, and the random baseline separates real knowledge from guessing.

What would settle it

Re-score every model's answers by hand, or with a judge whose error rate is measured per model, and compare the accuracy table; in particular, manually grading the 224 open numerical answers of the weakest models would reveal whether their 2–8 percent NUM scores are real or an artifact of the judge marking correct but messily formatted answers as incorrect.

Watch

Extended reading notes

Core claim

On the authors' measurements, Claude-3.5-Sonnet is the top model at 83.9±0.2 percent overall accuracy, with GPT-4o essentially tied at 83.8±1.3 percent; Claude-3-Opus (77.9), GPT-4-turbo (76.4), and GPT-4o-mini (70.5) follow, while GPT-4 sits at 60.8 percent. Among open-source models, Llama3-70b leads at 56.0 percent, Phi3-14b reaches 42.8 percent, and Phi3-3.8b reaches 36.5 percent; Llama2-7b, Llama2-70b, Llama3-8b, and Mistral-7b all fall below 32 percent, with Llama2-7b and Mistral-7b scoring below the random guessing baseline of 16.3 percent on parts of the test. The open-ended numerical questions (NUM) are the hardest category: even the best model, Claude-3.5-Sonnet, gets 74.6 percent there, while the weaker open models score in the low single digits. The paper interprets the results as a raw-capability baseline and as evidence that fine-tuning and prompt engineering have a large potential upside for open-source models.

Load-bearing premise

The accuracy table rests on the assumption that GPT-4o, as automated judge, classifies answers correctly for all fifteen models, but it was manually validated on only four models, and even with the final prompt it still misclassified 15 of Llama2-7b's 644 answers — errors that could bias the reported scores, especially for weak open-source models, and shift the ranking.

Editorial extensions

If this is right

Claude-3.5-Sonnet and GPT-4o set the current ceiling at roughly 84 percent, with GPT-4o strongest on multiple-choice questions and Claude-3.5-Sonnet strongest on open numerical ones.
The best open-source model, Llama3-70b, trails the closed-source leaders by about 28 points, quantifying the headroom that fine-tuning would need to close.
Open-ended numerical questions are the universal weak spot: every model scores lowest there, and the weakest models fall to 2–9 percent.
Several open-source models, including Llama2-7b and Mistral-7b, perform at or below random guessing in MATCH and MCQN categories, indicating systematic reasoning flaws rather than mere knowledge gaps.
Local hardware matters for local models: Llama3-8b improved by about 16 percent when moved from a MacBook M1 to a GPU server, while API-based models were unaffected.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the judge's errors are systematic against poorly formatted answers, the open-source scores are likely lower bounds rather than point estimates, and the gap between closed and open models may be smaller than reported.
The Phi3-3.8b result — 36.5 percent at 3.8 billion parameters, above several larger models — suggests parameter count is not the main driver, so fine-tuning a small open model may yield outsized returns at low hardware cost.
Because GATE questions are public exam materials, the closed-source models may have encountered them during pretraining, so the 84 percent figure could overstate genuine materials-science reasoning; a fresh question set would test transfer.
The NUM category could serve as a cheap diagnostic for tokenization and arithmetic quality when comparing future model versions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate and useful benchmark extension, and the headline ranking is probably right, but the paper overstates the precision of the open-source numbers. The GPT-4o judge was manually checked on only four models, and the known errors run one-way for weak models, so the bottom of the table carries more uncertainty than the error bars show.

What's actually new: Zaki et al. tested three models on MaScQA; this paper sweeps fifteen, adds a three-run repeat, a random baseline, per-category breakdowns, and a hardware comparison. They even ship the raw responses and code on GitHub. That's real empirical work, and the paper is honest about the judge problem: they report misclassification counts for the four models they checked, show examples of ambiguous Llama2-7b answers, and try a second, more detailed prompt that cuts errors from 48 to 15 out of 644. Credit where due.

The soft spots are not fatal but they matter. The judge was validated on four models; the other eleven are scored with no manual check. For Llama2-7b the final prompt still misclassifies 15 of 644 answers, and the errors are asymmetric—correct answers marked incorrect much more often than the reverse. Weak models produce ambiguous outputs, and an automated judge can systematically undercount them. So the absolute accuracies for Llama2-70b (28%), Phi3-3.8b (36%), and Phi3-14b (43%) are probably lower bounds, and the size of the "fine-tuning gap" they advertise is not pinned down. The ranking of closed versus open is robust; the specific numbers less so.

The hardware split is a smaller but real nuisance: some local models run on a MacBook M1, some on an 8×A100 box, and they report a 16% difference for Llama3-8b between the two. That makes cross-model comparisons a bit dirty, though it likely doesn't change the top-tier ordering. They also removed six benchmark questions ad hoc; the details are in the SI, but the paper should at least list them in the main text. And they never discuss training-data contamination: MaScQA is built from GATE, a public exam, so these models may have seen the exact questions. That's a known risk and should be addressed.

Minor internal issues: the abstract says GPT-4 where the body says GPT-4o, and one results paragraph says the top models exceeded ~70% just before the tables report 84%. These look like typos, but they'll confuse readers.

Bottom line: this is a good empirical baseline for anyone working on materials-science LLMs, and it deserves a serious referee. The right fix is not rejection; it's asking for judge validation on more model families, a transparent list of the removed questions, and a contamination discussion. I'd cite it as a comparison point and bring it to a reading group.

Recommendation: engage with it, send it to peer review, and push for the revisions above.

Referee Report

3 major / 4 minor

Summary. The paper evaluates 15 LLMs (closed- and open-source) on the MaScQA benchmark of 644 questions, using GPT-4o as an automated judge with a three-run protocol at temperature 0, and compares results to a random baseline and to the original Zaki et al. study. It reports Claude-3.5-Sonnet and GPT-4o as top performers at roughly 84% accuracy, Llama3-70b at 56.0%, Phi3-14b at 42.8%, and weaker open-source models far below. The authors frame the results as a baseline for fine-tuning an open-source materials-science LLM.

Significance. If the measurements are accurate, the paper provides a useful and reproducible baseline for LLM performance on a domain-specific benchmark, with open code and data, three repeated runs per question, and a transparent comparison to prior work. The central qualitative ranking (closed-source models outperform open-source ones) is plausible and broadly consistent with other LLM evaluations. However, the quantitative claims about open-source models, and hence the perceived fine-tuning gap, rest on validation of the automated judge on only four of fifteen models, so the absolute numbers should be treated with caution until the judge is validated on the remaining models.

major comments (3)

[§2.2, Tables 2–3] The GPT-4o judge is manually validated on only four of fifteen models, and the reported misclassifications are systematically one-directional for weaker models: for Llama2-7b, 48 of 644 correct answers were labeled incorrect under the first prompt, and 15 of 644 remain misclassified under the final prompt. The remaining eleven models, including all Phi3 models, Llama2-70b, Llama3-70b, Mistral-7b, and several Claude/GPT variants, are scored without any manual check. Because the paper's claims about open-source performance (Llama3-70b at 56.0%, Phi3-14b at 42.8%, Llama2-70b at 28.1%) and the size of the fine-tuning gap depend on these absolute numbers, the authors should manually score a random sample of responses for each unvalidated model and report corrected accuracies or a bias-adjusted confidence interval.
[§2.2.2, Figure 4(b)] The evaluation prompt instructs GPT-4o to judge an answer as correct only if the reasoning is sound, yet §2.2 states that for MATCH, MCQ, and MCQN questions the response is assessed solely by the selected letter. This contradiction means that answers with the correct option but flawed reasoning—precisely the ambiguous cases shown in Figure 3—may be marked incorrect, systematically lowering scores for weak models that produce verbose but incorrect explanations. The prompt should be aligned with the stated letter-based policy, or the policy should be revised and the judge revalidated under the actual scoring rule.
[§3, text versus Table 5] The prose misreports the numerical results in Table 5: it says Claude-3-Opus and GPT-4-turbo 'closely follow with ∼64−63%' when the table lists 77.9% and 76.4%, and it describes Llama3-70b as achieving results 'with ∼32.6%' when its overall accuracy is 56.0% (32.6% is its NUM-category score). These discrepancies make the results difficult to interpret and should be corrected.

minor comments (4)

[§1.2] The statement that manually removing six questions 'does not significantly bias the evaluation outcomes' is an assertion without supporting analysis; please provide the list of removed questions and a brief justification, or quantify the sensitivity.
[Tables 4–5] For the four models tested on both machines (GPT-4, GPT-3.5-turbo, Llama2-7b, Llama3-8b), Table 5 reports a single accuracy, but the paper does not state how the two machine-specific results are combined; please clarify whether the reported value is the average of the two runs or a pooled result.
[Figure 5] The caption says the figure includes 'the 15 unique LLMs tested' but the plotted quantities are per-category total correct answers; please clarify whether these are per-category totals averaged over all models, and label the axes accordingly.
[§2.3] Setting the random baseline for NUM questions to exactly 0% with zero standard deviation is a reasonable simplification, but in practice the accepted numerical range may be large enough that random guessing has a tiny nonzero probability; a brief note acknowledging this would be more precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported accuracies are external benchmark measurements; the GPT-4o judge validation gap is a measurement-validity risk, not a self-referential derivation.

full rationale

This paper is an empirical evaluation, not a derivation. Each reported accuracy is a count of correct answers obtained by comparing LLM outputs to the externally defined MaScQA/GATE benchmark answers. No parameter is fitted to those counts, and no central result is constructed from the measured quantities. The random baseline in Section 2.3 is computed independently from binomial probabilities and is used only for comparison, not to generate the model scores. The use of GPT-4o as an automated judge is a measurement instrument: its calibration was manually checked on four models in Section 2.2, and the paper's own Discussion explicitly acknowledges that lower-performing models may be disproportionately misclassified. That is a validity and measurement-error caveat, not circularity, because the judge's errors are not equivalent by construction to the reported accuracies; correcting any bias would change the numbers without making the claim definitionally true. The citation to Zaki et al. is a normal reference to the benchmark's origin, and the authors' removal of six questions from the dataset is a small data-cleaning edit, not a way of encoding the target conclusion. No load-bearing step of the analysis reduces to its own inputs, and no result is imported from a self-citation chain as a substitute for evidence. The central claim is self-contained against an external benchmark, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's central measurements rest on four external assumptions: the reliability of the automated judge across all models, the benign nature of the authors' 6-question removal, the validity of temperature-0 sampling, and the correctness of the benchmark ground truth. None of these are derived or tested beyond the limited manual validation reported in Section 2.2.

assumptions (4)

domain assumption GPT-4o judge accuracy observed on four models (about 97.9 percent) generalizes to all fifteen models.
The final accuracy numbers for all models are produced by the GPT-4o judge, but manual validation was performed only for Claude-3.5-Sonnet, GPT-4-turbo, Llama3-8b, and Llama2-7b (Table 3).
ad hoc to paper The six questions removed from MaScQA were removed for legitimate reasons and do not bias results.
Section 1.2 states the authors 'refined' the benchmark by removing 6 Q&A samples due to duplication or missing information, but the details are only promised in supplementary material not visible in the preprint.
domain assumption Temperature 0 plus three repeats gives a representative sample of each model's ability.
Section 2.1 argues temperature 0 yields the most probable answer, though the paper also notes residual stochasticity from MoE routing and floating point.
domain assumption The MaScQA ground truth labels are correct and complete.
The entire evaluation treats MaScQA as the external standard, relying on the benchmark authors' curation and the original GATE answer keys.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering." pith.science (2026). https://pith.science/paper/VSN6R735

@misc{pith2026250104277,
  author       = {Pith},
  title        = {Pith review of: Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSN6R735}},
  note         = {Machine review of arXiv:2501.04277}
}

read the original abstract

The integration of artificial intelligence into various domains is rapidly increasing, with Large Language Models (LLMs) becoming more prevalent in numerous applications. This work is included in an overall project which aims to train an LLM specifically in the field of materials science. To assess the impact of this specialized training, it is essential to establish the baseline performance of existing LLMs in materials science. In this study, we evaluated 15 different LLMs using the MaScQA question answering (Q&A) benchmark. This benchmark comprises questions from the Graduate Aptitude Test in Engineering (GATE), tailored to test models' capabilities in answering questions related to materials science and metallurgical engineering. Our results indicate that closed-source LLMs, such as Claude-3.5-Sonnet and GPT-4, perform the best with an overall accuracy of ~84%, while the open-source models, Llama3-70b and Phi3-14b, top at ~56% and ~43%, respectively. These findings provide a baseline for the raw capabilities of LLMs on Q&A tasks applied to materials science, and emphasize the substantial improvement that could be brought to open-source models via prompt engineering and fine-tuning strategies. We anticipate that this work could push the adoption of LLMs as valuable assistants in materials science, demonstrating their utility in this specialized domain and related sub-domains.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 9 canonical work pages

[1]

We used the OpenAI, Anthropic and Ollama APIs to access the models 24–26. The models used in this study are: GPT- 4-turbo, GPT-4o, GPT-4o-mini, GPT-4, GPT-3.5-turbo, Claude-3- Opus, Claude-3-Haiku, Claude-3.5-Sonnet, Llama2-7b, Llama2- 70b, Llama3-8b, Llama3-70b, Mistral-7b, Phi3-3.8b and Phi3- 14b. The tokenization process for all LLMs was handled auto- ...

work page 2020
[2]

Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering

In this study , we adopt the second strategy , leveraging the instruction-following capabilities and general NLP proficiency of pre-existing models. Our final objective is to fine-tune an exist- ing LLM and integrated it into a retrieval-augmented generation (RAG) system for materials science applications. To guide this fu- ture fine-tuning process and es...

work page Pith review arXiv 2025
[3]

as shown in Figure 4 a) and then submitted the question in the format:

This analysis demonstrates GPT-4o’s accuracy as a judge while also identifying areas where discrepancies arise, particularly for questions requiring nuanced reasoning. 2.2.1 First approach Initially , we selected GPT-4o for this task, using a straightforward prompt: "Based on the question and the correct answer, You must tell if the other answer is correc...

work page 2012
[9]

Chen, F.-K

16 Z.-Y. Chen, F.-K. Xie, M. Wan, Y. Yuan, M. Liu, Z.-G. Wang, S. Meng and Y.-G. Wang,Chinese Physics B, 2023, 32, 118104. 17 T. Xie, Y. Wan, W. Huang, Z. Yin, Y. Liu, S. Wang, Q. Linghu, C. Kit, C. Grazian, W. Zhang et al., arXiv preprint arXiv:2308.13565,

arXiv 2023
[10]

Chiang, C.-H

18 Y. Chiang, C.-H. Chou and J. Riebesell, arXiv preprint arXiv:2401.17244,

arXiv
[11]

19 Y. Song, S. Miret and B. Liu, arXiv preprint arXiv:2305.08264,

arXiv
[12]

Hendrycks, C

20 D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song and J. Steinhardt, Measuring Mathemati- cal Problem Solving With the MATH Dataset , 2021, https: //arxiv.org/abs/2103.03874. 21 M. Courbariaux, Y. Bengio and J.-P. David, ICLR (Workshop),

arXiv 2021
[14]

MCQ tasks, while simpler, can be impacted by pattern exploitation where models rely on super- ficial cues rather than true conceptual understanding

For MATCH tasks, LLMs struggle to establish logical relationships between entities, often default- ing to heuristic-based reasoning. MCQ tasks, while simpler, can be impacted by pattern exploitation where models rely on super- ficial cues rather than true conceptual understanding. These trends underscore the importance of prompt optimiza- tion and domain-...

work page

Show all 16 references

[15]

Brugger, S

23 C. Brugger, S. Weithoffer, C. De Schryver, U. Wasenmüller and N. Wehn, Advances in Radio science , 2014, 12, 75–81. 24 OpenAI, OpenAI API , https://openai.com/api/, 2023, Ac- cessed: May-Aug

2014
[16]

25 Anthropic, Anthropic API , https://www.anthropic.com, 2023, Accessed: May-Aug

2023
[17]

26 Ollama, Ollama API , https://ollama.ai, 2023, Accessed: May-Aug

2023
[327]

2 W. Lu, R. K. Luu and M. J. Buehler, arXiv preprint arXiv:2409.03444,

arXiv
[2015]

22 A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary , C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bres- sand et al., arXiv preprint arXiv:2401.04088 ,

arXiv
[2019]

6 Y. An, J. Greenberg, A. Kalinowski, X. Zhao, X. Hu, F. J. Uribe-Romo, K. Langlois, J. Furst and D. A. Gómez-Gualdrón, Knowledge Graph Question Answering for Materials Science (KGQA4MAT): Developing Natural Language Interface for Metal-Organic Frameworks Knowledge Graph (MOF-...

2024 arXiv
[2023]

Welbl, N

12 J. Welbl, N. F. Liu and M. Gardner, Proceedings of the 3rd Workshop on Noisy User-generated Text, Copenhagen, Den- mark, 2017, pp. 94–106. 13 D. Zhang, W. Liu, Q. Tan, J. Chen, H. Yan, Y. Yan, J. Li, W. Huang, X. Yue, D. Zhou, S. Zhang, M. Su, H.-S. Zhong, Y. Li and W. Ouya...

2017
[2024]

14 X. Lu, H. Cao, Z. Liu, S. Bai, L. Chen, Y. Yao, H.-T. Zheng and Y. Li, arXiv preprint arXiv:2403.08192 ,

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

We used the OpenAI, Anthropic and Ollama APIs to access the models 24–26. The models used in this study are: GPT- 4-turbo, GPT-4o, GPT-4o-mini, GPT-4, GPT-3.5-turbo, Claude-3- Opus, Claude-3-Haiku, Claude-3.5-Sonnet, Llama2-7b, Llama2- 70b, Llama3-8b, Llama3-70b, Mistral-7b, Phi3-3.8b and Phi3- 14b. The tokenization process for all LLMs was handled auto- ...

work page 2020

[2] [2]

Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering

In this study , we adopt the second strategy , leveraging the instruction-following capabilities and general NLP proficiency of pre-existing models. Our final objective is to fine-tune an exist- ing LLM and integrated it into a retrieval-augmented generation (RAG) system for materials science applications. To guide this fu- ture fine-tuning process and es...

work page Pith review arXiv 2025

[3] [3]

as shown in Figure 4 a) and then submitted the question in the format:

This analysis demonstrates GPT-4o’s accuracy as a judge while also identifying areas where discrepancies arise, particularly for questions requiring nuanced reasoning. 2.2.1 First approach Initially , we selected GPT-4o for this task, using a straightforward prompt: "Based on the question and the correct answer, You must tell if the other answer is correc...

work page 2012

[4] [9]

Chen, F.-K

16 Z.-Y. Chen, F.-K. Xie, M. Wan, Y. Yuan, M. Liu, Z.-G. Wang, S. Meng and Y.-G. Wang,Chinese Physics B, 2023, 32, 118104. 17 T. Xie, Y. Wan, W. Huang, Z. Yin, Y. Liu, S. Wang, Q. Linghu, C. Kit, C. Grazian, W. Zhang et al., arXiv preprint arXiv:2308.13565,

arXiv 2023

[5] [10]

Chiang, C.-H

18 Y. Chiang, C.-H. Chou and J. Riebesell, arXiv preprint arXiv:2401.17244,

arXiv

[6] [11]

19 Y. Song, S. Miret and B. Liu, arXiv preprint arXiv:2305.08264,

arXiv

[7] [12]

Hendrycks, C

20 D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song and J. Steinhardt, Measuring Mathemati- cal Problem Solving With the MATH Dataset , 2021, https: //arxiv.org/abs/2103.03874. 21 M. Courbariaux, Y. Bengio and J.-P. David, ICLR (Workshop),

arXiv 2021

[8] [14]

MCQ tasks, while simpler, can be impacted by pattern exploitation where models rely on super- ficial cues rather than true conceptual understanding

For MATCH tasks, LLMs struggle to establish logical relationships between entities, often default- ing to heuristic-based reasoning. MCQ tasks, while simpler, can be impacted by pattern exploitation where models rely on super- ficial cues rather than true conceptual understanding. These trends underscore the importance of prompt optimiza- tion and domain-...

work page