REVIEW 5 major objections 5 minor 57 references
What do Large Language Models know about materials?
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Open-weights LLMs can answer simple materials facts from memory, yet they confidently invent melting points for elements that have none.
desk verdict A useful tokenization analysis and an honest but provisional size-accuracy claim; worth a round of revisions before the central trend can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a minimal, reproducible benchmark: a fixed in-context prompt ("Pi is 3.14<. The melting temperature of ice in Kelvin is 273.15K<.") that teaches the model the number format, the unit, and a custom end-of-sequence marker; greedy argmax generation capped at 20 tokens; and regular-expression parsing of the output against melting-temperature data from the mendeleev package. The prompt exploits in-context learning so that model output is easy to parse unambiguously, while the vocabulary inspection counts how many tokens are needed to represent each element name in each model's tokenizer.
What would settle it
Run the melting-temperature benchmark on the same models with two prompts that are identical except the in-context example changes the magnitude, e.g., replacing "273.15K" with "300K". If model outputs shift toward the injected value, the reported accuracy reflects prompt-driven imitation rather than intrinsic knowledge.
Extended reading notes
Core claim
The paper claims that the intrinsic material knowledge of base LLMs, measured by their ability to reproduce melting temperatures from the Periodic Table, improves with model size in the range from 1 to 3 billion parameters, with the largest tested model generating correct values for many elements. It also claims that this knowledge is shaped by the vocabulary: element names that are single tokens are more reliably generated and embedded, whereas subword-tokenized names require attention-based composition and are more fragile. The hallucination problem is shown to be independent of this size trend: every model assigns a melting temperature to helium and to short-lived elements where no experimental value exists, so accuracy on existing values does not imply correctness about nonexistent ones. The paper presents this as a starting point for a benchmark that decides for which steps of the Processing-Structure-Property-Performance chain LLMs can be used and where specialized models are required.
Load-bearing premise
The entire accuracy comparison rests on the particular in-context prompt and parsing rules being an unbiased probe of what the models know, and the paper reports no control experiments with alternative prompts or parsing rules.
Editorial extensions
If this is right
- If the size trend holds, larger base LLMs become a viable first-pass source for simple material property queries, and even a 3-billion-parameter model can supply correct values for common elements.
- Because hallucinations persist across sizes, any LLM-based engineering workflow must include a validation or sanity-check layer before numerical values are used.
- The tokenization analysis implies that specialized, fine-tuned materials models should reserve single vocabulary tokens for material fingerprints and train those embeddings on enough in-domain data.
- The same benchmark procedure can be applied to all properties in the mendeleev package and to any open-weights model, producing a foundation-model ranking for downstream fine-tuning.
- The approach can be extended step by step along the PSPP chain, identifying which transitions LLMs can handle directly and which still need physics-based or specialized models.
Reading between the lines
- The reported accuracy likely conflates in-context-learning compliance with true memorial knowledge: a larger model may simply follow the prompt's format better, and the size trend could partly be a format-following trend rather than a knowledge trend.
- A concrete probe would be to inject a deliberately wrong in-context example, such as "The melting temperature of ice is 300K"; if outputs shift toward 300K, the benchmark is measuring imitation of the prompt rather than retrieval of stored facts.
- Melting temperatures are unusually well represented on the public internet, so the accuracy measured here probably overstates what the same models would achieve for rare alloys, processing parameters, or performance data further along the PSPP chain.
- The benchmark's usefulness for model choice would improve if it reported per-property accuracy separately, since a single aggregate score could hide strong performance on common properties and weak performance on niche ones.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates what factual materials knowledge is encoded in open-weights base LLMs, using the Periodic Table of Elements as a ground-truth testbed. Section 3.1 analyzes tokenization uniqueness of element names across ten models, relating token counts to the design of material fingerprints and embeddings. Section 3.2 probes the melting temperatures of pure elements with a fixed prompt, a 20-token cap, argmax sampling, and regular-expression-based value extraction. Based on three models with 1, 2, and 3 billion parameters, the authors claim that larger models reproduce known melting temperatures better, while all models hallucinate melting points for helium and short-lived elements. The paper positions itself as a starting point for a benchmark on end-to-end materials tasks and for informed model selection in the Processing-Structure-Property-Performance chain.
Significance. If fully substantiated, the paper would provide a useful, low-cost procedure for sanity-checking whether base LLMs can serve as material property lookup tools, and it would draw attention to a real risk: even small models can reproduce simple facts, but hallucinations on non-existent or undetermined values require validation layers. The evaluation is clean in one important respect: the ground truth is the independent mendeleev package, and no fitted parameters or back-fitted prompts are involved, so there is no circularity in the accuracy target. The tokenization analysis is also transparent and easily reproducible. However, the central empirical claim currently rests on visual inspection of three models from different families under one engineered prompt, with no quantitative accuracy metric, no tolerance definition, no parser specification, and no prompt or parser control experiments. The paper's value as a benchmark therefore depends on revisions that turn the reported qualitative trends into measurable, reproducible quantities.
major comments (5)
- [§3.2, Figure 4] The central claim that larger models give better reproduction of melting temperatures is supported only by visual inspection of three colored periodic tables. No quantitative accuracy rate, no tolerance window, and no per-element classification are reported. Please define a tolerance (for example, relative or absolute deviation from the mendeleev value), report the fraction of elements judged correct per model, and give false-positive and false-negative counts. Without such numbers, the statement 'bigger models also enable a better reproduction of values' is not a measurable result.
- [§3.2, custom prompt] The probe prompt is 'Pi is 3.14<. The melting temperature of ice in Kelvin is 273.15K<.', which supplies an in-context example of the exact target property, a unit, a number format, and a custom end-of-sequence marker. The paper states that 'preliminary studies' selected this prompt but provides no control experiments with alternative prompts (for example, no in-context example, different units, an explicit 'If no value exists, provide NaN' instruction, or no custom end marker) and no sensitivity analysis of the regular-expression parser. The accuracy and hallucination trends could therefore reflect prompt adherence and parser behavior rather than intrinsic factual knowledge. The authors' own admission that adding 'If no value exists, provide NaN' reduces false values confirms that the measured hallucination rate is prompt-dependent.
- [§3.2 and Table 1] The size trend is confounded by model family: the three models compared in Figure 4 are gemma-3-1b-pt, gemma-2-2b, and Llama-3.2-3B, which differ in architecture, vocabulary size, tokenizer, and training data. Table 1 includes same-family size series (gemma-3-1b-pt vs gemma-3-4b-pt and Qwen3-0.6B/1.7B/4B) that are not shown in Figure 4. Without at least one same-family size comparison, the conclusion 'bigger is better' cannot be separated from the effect of model family.
- [§3.2, hallucination claim] The statement that a melting temperature of helium is 'always predicted' is based on three models and one prompt, and the paper immediately notes that prompting can reduce such false values. This is anecdotal rather than a measured result. Please quantify the hallucination rate as a function of model, prompt variant, and element, and report the number of nonsensical melting points before and after the NaN instruction. As written, the robustness of the hallucination observation is not established.
- [§3.2 and 'Overview of the applied models'] The paper promises that the authors 'provide the code for benchmarking' but no repository, DOI, or exact parsing rules are given. The evaluation depends critically on the regular-expression extraction and unit conversion, yet these are not specified. Please include the parser grammar, the exact prompt template, the sampling parameters, and a link to the code and data. This is necessary for the claimed benchmark to be reproducible and for the results to be independently verifiable.
minor comments (5)
- [§3.1] The claim that common uses such as 'iron will' and 'silver bullet' shift the embeddings of chemical element names into non-scientific dimensions is speculative; the paper presents no embedding-similarity or attention analysis. Please either add a small quantitative check or soften the wording to a hypothesis.
- [Figure 4] The color maps in Figure 4 have no legend or colorbar, and it is unclear whether 'truthful' means exact string match, a numerical tolerance, or a unit-converted match. Adding a legend and stating the matching criterion in the caption would make the visual claims interpretable.
- [Introduction and §2.2] There are several typographical errors, including 'Eucledian distance' and 'subword tokenziation'; please copyedit the manuscript.
- [§4] The final paragraph ends with the stray fragment 'e au-thor.', which appears to be a formatting artifact; it should be removed or completed.
- [Conclusion and abstract] The paper calls itself 'a starting point of building a benchmark,' but the abstract and conclusion also describe it as leading to 'a material knowledge benchmark.' This overstates what is delivered; consider using language that clearly distinguishes the proposed framework from a validated benchmark.
Circularity Check
No significant circularity: LLM outputs are compared against the independent mendeleev database, and the authors' self-citations are background only.
full rationale
The paper's main empirical claim is an experimental comparison: LLM-generated melting temperatures are checked against the mendeleev package [51], an independent external reference, and token counts are taken directly from each model's vocabulary. There is no derivation chain in which an input is defined in terms of an output. The custom prompt in Section 3.2 ('Pi is 3.14<. The melting temperature of ice in Kelvin is 273.15K<.') is a fixed few-shot format example, not a per-element fitted value; it is not used to compute the reported accuracy, and the outputs for individual elements are generated by the models and then compared with external data. The observation that a helium melting point is hallucinated and that adding 'If no value exists, provide NaN' reduces false outputs is an empirical prompt-sensitivity result, not a prediction derived from its own input. The authors' self-citations ([4], [6], [38], [42], [43], [44]) appear as illustrative background about hydrogels and SHAPES structures and do not justify the benchmark's correctness. Methodological limitations—one fixed prompt, no quantitative accuracy metric, and no parser-control experiments—are validity concerns, not circularity, because the benchmark remains anchored to an independent external ground truth. Therefore no specific circular step can be exhibited.
Assumptions & free parameters
assumptions (4)
- domain assumption Melting temperatures from the mendeleev Python package form a correct and complete ground truth for elemental materials.
- domain assumption A single-token representation of an element name implies a more specialized embedding and better material-fingerprint uniqueness.
- ad hoc to paper The custom in-context prompt ('Pi is 3.14<. The melting temperature of ice in Kelvin is 273.15K<.') with argmax sampling and a 20-token cap yields a valid, unbiased probe of intrinsic factual knowledge.
- domain assumption Base, non-instruction-tuned models are the appropriate object of study for PSPP-chain material knowledge.
Cite this review
Pith. "Pith review of What do Large Language Models know about materials?." pith.science (2026). https://pith.science/paper/7H3RYRYL
@misc{pith2026250714586,
author = {Pith},
title = {Pith review of: What do Large Language Models know about materials?},
year = {2026},
howpublished = {\url{https://pith.science/paper/7H3RYRYL}},
note = {Machine review of arXiv:2507.14586}
}
read the original abstract
Large Language Models (LLMs) are increasingly applied in the fields of mechanical engineering and materials science. As models that establish connections through the interface of language, LLMs can be applied for step-wise reasoning through the Processing-Structure-Property-Performance chain of material science and engineering. Current LLMs are built for adequately representing a dataset, which is the most part of the accessible internet. However, the internet mostly contains non-scientific content. If LLMs should be applied for engineering purposes, it is valuable to investigate models for their intrinsic knowledge -- here: the capacity to generate correct information about materials. In the current work, for the example of the Periodic Table of Elements, we highlight the role of vocabulary and tokenization for the uniqueness of material fingerprints, and the LLMs' capabilities of generating factually correct output of different state-of-the-art open models. This leads to a material knowledge benchmark for an informed choice, for which steps in the PSPP chain LLMs are applicable, and where specialized models are required.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Advances in neural information processing systems 2017, 30
work page 2017
-
[2]
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, C. Zhang, In The Eleventh International Conference on Learning Representations. 2022
work page 2022
-
[3]
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al., arXiv preprint arXiv:2101.00027 2020
arXiv 2020
-
[4]
A. Ehrenhofer, T. Wallmersperger, In Proceedings of the ECCOMAS Smart Conference . 2025
work page 2025
- [5]
-
[6]
Y. Wang, T. Wallmersperger, A. Ehrenhofer, Engineering Reports 2024, 6, 11 e12893
work page 2024
-
[7]
S. Curtarolo, G. L. Hart, M. B. Nardelli, N. Mingo, S. Sanvito, O. Levy, Nature materials 2013, 12, 3 191
work page 2013
-
[8]
K. Choudhary, B. DeCost, C. Chen, A. Jain, F. Tavazza, R. Cohn, C. W. Park, A. Choudhary, A. Agrawal, S. J. Billinge, et al., npj Computational Materials 2022, 8, 1 59
work page 2022
Show all 57 references
-
[9]
Ramakrishna, T.-Y
S. Ramakrishna, T.-Y. Zhang, W.-C. Lu, Q. Qian, J. S. C. Low, J. H. R. Yune, D. Z. L. Tan, S. Bressan, S. Sanvito, S. R. Kalidindi, Journal of Intelligent Manufacturing 2019, 30 2307. 12
2019
-
[10]
T. Zhou, Z. Song, K. Sundmacher, Engineering 2019, 5, 6 1017
2019
-
[11]
Ohana, M
R. Ohana, M. McCabe, L. Meyer, R. Morel, F. Agocs, M. Beneitez, M. Berger, B. Burkhart, S. Dalziel, D. Fielding, et al., Advances in Neural Information Processing Systems 2024, 37 44989
2024
-
[12]
D. S. Levine, M. Shuaibi, E. W. C. Spotte-Smith, M. G. Taylor, M. R. Hasyim, K. Michel, I. Bata- tia, G. Cs´ anyi, M. Dzamba, P. Eastman, et al.,arXiv preprint arXiv:2505.08762 2025
2025
-
[13]
S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. Kocher, S. Cholia, D. Gunter, V. L. Chevrier, K. A. Persson, G. Ceder, Computational Materials Science 2013, 68 314
2013
-
[14]
T. D. Huan, A. Mannodi-Kanakkithodi, C. Kim, V. Sharma, G. Pilania, R. Ramprasad, Scientific data 2016, 3, 1 1
2016
-
[15]
Curtarolo, W
S. Curtarolo, W. Setyawan, G. L. Hart, M. Jahnatek, R. V. Chepulskii, R. H. Taylor, S. Wang, J. Xue, K. Yang, O. Levy, et al., Computational Materials Science 2012, 58 218
2012
-
[16]
Sarker, T
P. Sarker, T. Harrington, C. Toher, C. Oses, M. Samiee, J.-P. Maria, D. W. Brenner, K. S. Vecchio, S. Curtarolo, Nature communications 2018, 9, 1 4980
2018
-
[17]
Doan Tran, C
H. Doan Tran, C. Kim, L. Chen, A. Chandrasekaran, R. Batra, S. Venkatram, D. Kamal, J. P. Lightstone, R. Gurnani, P. Shetty, et al., Journal of Applied Physics 2020, 128, 17
2020
-
[18]
Kuenneth, A
C. Kuenneth, A. C. Rajan, H. Tran, L. Chen, C. Kim, R. Ramprasad, Patterns 2021, 2, 4
2021
-
[19]
Jumper, R
J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. ˇZ ´ ıdek, A. Potapenko, et al.,nature 2021, 596, 7873 583
2021
-
[20]
B. Yin, K. M. Liew, Composite Structures 2021, 273 114328
2021
-
[21]
N. J. Szymanski, B. Rendy, Y. Fei, R. E. Kumar, T. He, D. Milsted, M. J. McDermott, M. Gallant, E. D. Cubuk, A. Merchant, et al., Nature 2023, 624, 7990 86
2023
-
[22]
T. D. Huan, A. Mannodi-Kanakkithodi, R. Ramprasad, Physical Review B 2015, 92, 1 014106
2015
-
[23]
Ramprasad, R
R. Ramprasad, R. Batra, G. Pilania, A. Mannodi-Kanakkithodi, C. Kim, npj Computational Mate- rials 2017, 3, 1 54
2017
-
[24]
H. Xu, Y. Yu, Z. Wang, G. Shao, Energy & Environmental Materials 2019, 2, 4 234
2019
-
[25]
A. D. White, Nature Reviews Chemistry 2023, 7, 7 457
2023
-
[26]
K. M. Jablonka, Q. Ai, A. Al-Feghali, S. Badhwar, J. D. Bocarsly, A. M. Bran, S. Bringuier, L. C. Brinson, K. Choudhary, D. Circi, et al., Digital discovery 2023, 2, 5 1233
2023
-
[27]
Zhang, W
D. Zhang, W. Liu, Q. Tan, J. Chen, H. Yan, Y. Yan, J. Li, W. Huang, X. Yue, W. Ouyang, et al., arXiv preprint arXiv:2402.06852 2024
2024 arXiv
-
[28]
K. M. Jablonka, P. Schwaller, A. Ortega-Guerrero, B. Smit, Nature Machine Intelligence 2024, 6, 2 161
2024
-
[29]
Scherbakov, N
D. Scherbakov, N. Hubig, V. Jansari, A. Bakumenko, L. A. Lenert, Journal of the American Medi- cal Informatics Association 2025, 32, 6 1071
2025
-
[30]
Kononova, H
O. Kononova, H. Huo, T. He, Z. Rong, T. Botari, W. Sun, V. Tshitoyan, G. Ceder, Scientific data 2019, 6, 1 203
2019
-
[31]
Dagdelen, A
J. Dagdelen, A. Dunn, S. Lee, N. Walker, A. S. Rosen, G. Ceder, K. A. Persson, A. Jain, Nature Communications 2024, 15, 1 1418
2024
-
[32]
O. N. Oliveira Jr, M. C. F. Oliveira, Frontiers in chemistry 2022, 10 930369. 13
2022
-
[33]
X. Bai, S. He, Y. Li, Y. Xie, X. Zhang, W. Du, J.-R. Li, npj Computational Materials 2025, 11, 1 51
2025
-
[34]
Patki, R
N. Patki, R. Wedge, K. Veeramachaneni, In 2016 IEEE international conference on data science and advanced analytics (DSAA) . IEEE, 2016 399–410
2016
-
[35]
X. Wang, Z. Chen, H. Wang, L. Hou U, Z. Li, W. Guo, Data Science and Engineering 2025, 1–24
2025
-
[36]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Advances in neural information processing systems 2022, 35 24824
2022
-
[37]
Brown, B
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Advances in neural information processing systems 2020, 33 1877
2020
-
[38]
Ehrenhofer, T
A. Ehrenhofer, T. Wallmersperger, SHAPES Tables: A living document with a collection of sources for active materials and concepts of soft-hard active-passive embedded structures (SHAPES), 2025, URL https://zenodo.org/records/15000565
2025
-
[39]
J. Thik, S. Wang, C. Wang, H. Mansourifar, H. Lin, K. Okubo, C. Ling, J. Mater. Chem. A 2023, 11 25849
2023
-
[40]
Mavrogiannis, C
A. Mavrogiannis, C. Mavrogiannis, Y. Aloimonos, In 2024 IEEE International Conference on Robotics and Automation (ICRA) . 2024 17679–17686
2024
-
[41]
Z. Tan, C. Zhao, R. Moraffah, Y. Li, S. Wang, J. Li, T. Chen, H. Liu, arXiv preprint arXiv:2406.19417 2024
2024 arXiv
-
[42]
Ehrenhofer, T
A. Ehrenhofer, T. Wallmersperger, Smart Materials and Structures 2025, 34, 3 033005
2025
-
[43]
Ehrenhofer, A
A. Ehrenhofer, A. Mieting, S. Pfeil, J. Mersch, C. Cherif, G. Gerlach, T. Wallmersperger, In Y. Bar- Cohen, editor, Proc. SPIE 11587, Electroactive Polymer Actuators and Devices (EAPAD) XXII , volume 11375. International Society for Optics and Photonics, SPIE, 2020 47 – 58
2020
-
[44]
Binder, A
S. Binder, A. Ehrenhofer, A. Mieting, Stimulus-responsive Mesh, Patent No. US17/329,371, 2021, Submitted on 25 May 2021
2021
-
[45]
Devlin, M.-W
J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol- ume 1 (long and short papers) . 2019 4171–4186
2019
-
[46]
Huang, W
L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al., ACM Transactions on Information Systems 2025, 43, 2 1
2025
-
[47]
B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer, et al., In NeurIPS. 2023
2023
-
[48]
Mendelejew, Zeitschrift f¨ ur Chemie1869, 12, 5 405
D. Mendelejew, Zeitschrift f¨ ur Chemie1869, 12, 5 405
-
[49]
Schwerdtfeger, O
P. Schwerdtfeger, O. R. Smits, P. Pyykk¨ o,Nature reviews chemistry 2020, 4, 7 359
2020
-
[50]
Sheehan, Chemistry 1976, 49, 3 17
W. Sheehan, Chemistry 1976, 49, 3 17
1976
-
[51]
Mentel, URL https://github
L. Mentel, URL https://github. com/lmmentel/mendeleev 2021
2021
-
[52]
Kamath, J
A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram´ e, M. Rivi` ere, et al.,arXiv preprint arXiv:2503.19786 2025
2025 arXiv
-
[53]
Riviere, S
M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahri- ari, A. Ram´ e, et al.,arXiv preprint arXiv:2408.00118 2024. 14
2024 arXiv
-
[54]
Abdin, J
M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al., arXiv preprint arXiv:2412.08905 2024
2024 arXiv
-
[55]
Dubey, A
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., arXiv e-prints 2024, arXiv–2407
2024
-
[56]
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al., arXiv preprint arXiv:2505.09388 2025
2025 arXiv
-
[57]
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. 15 A Additional model results...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.