REVIEW 3 major objections 5 minor 58 references
Thermodynamic Prediction Enabled by Automatic Dataset Building and Machine Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read An LLM-powered extraction pipeline can build thermodynamic datasets from legacy papers, and the CatBoost model trained on them predicts mineral formation enthalpies with a mean absolute error of 46.5 kJ/mol.
desk verdict Useful data-mining pipeline, but the ML prediction claim is inflated by identity leakage from the random split and mineral-type features. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is LMExt plus CatBoost. LMExt, the Language Model Extractor, converts PDF pages into high-resolution images, OCRs them into two markdown files (full text and tables only), then submits each to an LLM with few-shot prompts that require evidence for every extracted value. The two-pass table/text split and the evidence requirement are what make extraction work on messy legacy documents. CatBoost then trains on the resulting dataset, using element-property features and one-hot encoded mineral types to predict $\Delta H^\circ_f$.
What would settle it
Retrain the CatBoost model after removing every data point whose label came from a pre-2000 LMExt extraction, then test on independently measured enthalpies; if the MAE stays near 46.5 kJ/mol the claim survives, while a large error increase shows the model relied on biased labels.
Extended reading notes
Core claim
The paper's central claim is that a fully automatic pipeline can go from a pile of legacy PDFs to a trained thermodynamic predictor. The LMExt pipeline first renders pages at 400 DPI and OCRs them into markdown, separately exposing tables, then prompts an LLM to extract values such as $\Delta H^\circ_f$ while justifying each choice. Applied to rare-earth-element minerals, this produced a dataset of 3185 entries. On an 80/20 split, a CatBoost model predicts $\Delta H^\circ_f$ with mean absolute error 46.5 kJ/mol, and 50.5% of test predictions are within 15 kJ/mol—comparable to typical experimental uncertainty. The paper presents this as evidence that LLM-based dataset construction can remove the manual-data bottleneck in ML-driven thermodynamics.
Load-bearing premise
The weakest load-bearing premise is that the combined manual-plus-LMExt thermodynamic dataset is accurate enough to serve as ground truth; LMExt's extraction accuracy on pre-2000 papers is only 43.8%, and if those errors are systematic the trained model inherits biases that the test split cannot reveal.
Editorial extensions
If this is right
- The pipeline turns a manual literature review that would take weeks into a few days of mostly automated extraction, so thermodynamic databases can grow faster than manual curation allows.
- Predictions with errors comparable to experimental scatter can serve as a pre-experimental reference, guiding which minerals are worth synthesizing and measuring.
- The evidence-prompting trick transfers: asking the LLM to justify each extraction materially improves success on legacy papers.
- Because the extraction tool is field-agnostic, the same workflow can build structured datasets for medical or financial documents, not just chemistry.
- Adding more curated data should push the model's accuracy further, since CatBoost handles small datasets and improves as the label set grows.
Reading between the lines
- The reported MAE is probably a lower bound on true error because the test labels themselves contain extraction noise; a model trained on fully cleaned labels would likely show a larger but more honest error.
- A useful next step is uncertainty quantification: flag predictions for mineral classes whose labels came mostly from pre-2000 extractions, so users know which values to trust.
- The evidence-prompting mechanism resembles chain-of-thought reasoning and could be pushed further with self-consistency voting or a separate verification pass to lift the 43.8% pre-2000 extraction rate.
- The dataset and model together form a benchmark that future LLM-based text-mining pipelines for chemistry can be measured against.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports two connected contributions: (1) LMExt, an LLM-based text data mining pipeline that converts PDF literature into machine-readable thermodynamic datasets, and (2) a CatBoost regression model trained on the resulting rare-earth-element (REE) mineral dataset to predict standard enthalpy of formation. The authors report near-perfect extraction accuracy for IUPAC stability constants, lower accuracy for pre-2000 thermodynamic papers (43.8% vs. 84.2% post-2000), and a test-set MAE of 46.5 kJ/mol for the CatBoost model, with 50.5% of test predictions within 15 kJ/mol. The paper also compares several LLMs on five selected papers and demonstrates LMExt on medical and financial documents.
Significance. If the claims hold, the work would be useful in two ways: it provides a practical recipe for LLM-based literature mining with evidence-prompting to handle legacy OCR-degraded documents, and it demonstrates a supervised model for mineral formation enthalpies built from automatically curated data. The paper is commendably concrete about the data sources and makes data and code links available. The extraction accuracy numbers, especially the pre-2000 failure rate, are honestly reported. However, the central quantitative claim of the ML section—that the model predicts enthalpies of minerals not in the training set—is not supported by the current evaluation protocol because the random split and one-hot mineral-type features permit identity leakage. The dataset-quality issues from the 43.8% pre-2000 extraction accuracy further weaken the label-grounding assumption. The strengths are the systematic workflow, the comparison across LLMs, and the clear statement of limitations, but the ML evaluation needs substantial revision.
major comments (3)
- [Results and Discussion, ML-based Thermodynamic Property Prediction; Method, Training ML model] The reported test MAE of 46.5 kJ/mol does not establish generalization to unseen minerals because the evaluation uses a random 80/20 split with no grouping by mineral species or chemical formula, while the input features include one-hot encoded general and specific mineral types. If the same mineral species appears in both training and test splits, CatBoost can memorize a mineral-specific average enthalpy, so the test error measures recall of known mineral identities rather than prediction for new compositions. The authors should split by mineral species/formula groups, report grouped cross-validation, and provide confidence intervals across multiple random seeds. This is load-bearing for the central claim that the model predicts minerals 'not included in its training set.'
- [Table 1 and Results and Discussion, Data Mining by LMExt] The training dataset is described as a mixture of manual review and LMExt output, and LMExt's extraction accuracy for pre-2000 thermodynamic papers is only 43.8%. If extraction errors are systematic (e.g., unit conversion mistakes, misassigned formulas, or OCR-induced value substitutions), the supervised model inherits those biases, and a held-out test split cannot reveal them because the test labels come from the same noisy pipeline. The authors should quantify label noise, report model performance separately on manually verified and LMExt-only entries, and assess sensitivity to plausible label errors. This is essential for interpreting the 46.5 kJ/mol MAE as a prediction of physical ground truth rather than of the pipeline's output.
- [Table 1 and Table 3] The extraction-accuracy evaluation rests on small samples: 50 randomly selected pairs for stability constants and five selected papers for the LLM comparison. These samples are too small to support the strong claims of '100% accuracy' and the ranking of LLMs, and no confidence intervals or per-paper breakdowns are given for the F1 and composite scores. The authors should report the per-paper results, the number of data points underlying each score, and an uncertainty estimate, or temper the claims accordingly.
minor comments (5)
- [Introduction and throughout] There are several typos and language issues, including 'thermondynamics' in the Introduction, 'ORC result' in the Data Mining section, 'successful rate' in the Data Mining section, and 'as Gemini 1.5-Pro' in the LLM comparison paragraph; these should be corrected.
- [Author contributions] The author contributions list 'A.H.' but the author list uses 'Haydn Anderson'; the initials should be made consistent.
- [Results and Discussion, Data Mining by LMExt] The claim that LMExt processes 'thousands of pages' by a single person in a few days would be more useful with a quantitative throughput estimate, such as pages per hour and cost per page, especially since the authors note that reasoning models are expensive; this would help readers assess practical applicability.
- [Method, Text data mining code] The statement that the code cannot be shown for intellectual property reasons is understandable, but the paper should clarify exactly which components are proprietary and which are available at the provided repository, so that reproducibility claims are precise.
- [Results and Discussion, ML-based Thermodynamic Property Prediction] The sentence 'the trained model successfully predicted ΔH°f of REE minerals within the test dataset' should be rephrased: a test-set evaluation shows correlation, not success in predicting 'minerals not included in its training set' unless a grouped split is used.
Circularity Check
No significant circularity: the ML prediction is trained on a literature-derived dataset with a held-out test split, and self-citations are contextual rather than load-bearing.
full rationale
The paper's central ML claim—CatBoost achieving a test MAE of 46.5 kJ/mol for standard enthalpy of formation—is a standard supervised regression result. The target values are held out in an 80/20 random split, and the input features (mineral type, chemical formula characteristics, constituent elements, atoms per formula unit) do not include the target enthalpy value, so no equation or fitted parameter is equivalent to the label by construction. The training dataset is cited to the authors' Thermochemical Database (reference 26), but that database is a literature-derived compilation and the model does not generate, define, or fit the labels; the self-citation is therefore contextual rather than load-bearing. The LMExt extraction-accuracy numbers (43.8% pre-2000, 84.2% post-2000, 100% for the IUPAC stability-constant sample) are evaluated against manually inspected ground truth, which is an external check rather than a circular reduction. The skeptic's concern that a random split with one-hot encoded mineral-type features may allow identity memorization is a legitimate generalization-risk critique, but it is not circularity: test values are not used as inputs, and the reported error is not forced by construction. No load-bearing uniqueness claim is imported from prior work, and no derivation step reduces to its own input. Score 0.
Assumptions & free parameters
free parameters (1)
- CatBoost hyperparameters =
Table S1 (not visible in preprint)
assumptions (3)
- domain assumption Literature thermodynamic values are accurate ground truths.
- domain assumption OCR output from Mistral at 400 DPI is a faithful text representation of the source PDFs.
- domain assumption Feature engineering via Mendeleev captures chemistry relevant to formation enthalpy.
Cite this review
Pith. "Pith review of Thermodynamic Prediction Enabled by Automatic Dataset Building and Machine Learning." pith.science (2026). https://pith.science/paper/BQQOWBXT
@misc{pith2026250707293,
author = {Pith},
title = {Pith review of: Thermodynamic Prediction Enabled by Automatic Dataset Building and Machine Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/BQQOWBXT}},
note = {Machine review of arXiv:2507.07293}
}
read the original abstract
New discoveries in chemistry and materials science, with increasingly expanding volume of requisite knowledge and experimental workload, provide unique opportunities for machine learning (ML) to take critical roles in accelerating research efficiency. Here, we demonstrate (1) the use of large language models (LLMs) for automated literature reviews, and (2) the training of an ML model to predict chemical knowledge (thermodynamic parameters). Our LLM-based literature review tool (LMExt) successfully extracted chemical information and beyond into a machine-readable structure, including stability constants for metal cation-ligand interactions, thermodynamic properties, and other broader data types (medical research papers, and financial reports), effectively overcoming the challenges inherent in each domain. Using the autonomous acquisition of thermodynamic data, an ML model was trained using the CatBoost algorithm for accurately predicting thermodynamic parameters (e.g., enthalpy of formation) of minerals. This work highlights the transformative potential of integrated ML approaches to reshape chemistry and materials science research.
Figures
Reference graph
Works this paper leans on
-
[1]
A. M. Schweidtmann, E. Esche, A. Fischer, M. Kloft, J.-U. Repke, S. Sager and A. Mitsos, Chemie Ingenieur Technik, 2021, 93, 2029-2039
work page 2021
-
[2]
M. Seifrid, R. Pollice, A. Aguilar-Granda, Z. Morgan Chan, K. Hotta, C. T. Ser, J. Vestfrid, T. C. Wu and A. Aspuru-Guzik, Accounts of Chemical Research, 2022, 55, 2454-2466
work page 2022
-
[3]
C. Lv, X. Zhou, L. Zhong, C. Yan, M. Srinivasan, Z. W. Seh, C. Liu, H. Pan, S. Li, Y . Wen and Q. Yan, Advanced Materials, 2022, 34, 2101474
work page 2022
- [4]
-
[5]
Pilania, Computational Materials Science, 2021, 193, 110360
G. Pilania, Computational Materials Science, 2021, 193, 110360
work page 2021
-
[6]
J. A. Keith, V . Vassilev-Galindo, B. Cheng, S. Chmiela, M. Gastegger, K.-R. Müller and A. Tkatchenko, Chemical Reviews, 2021, 121, 9816-9872
work page 2021
-
[7]
C. A. Meza Ramirez, G. Michael, A. Lorna and I. u. and Rehman, Applied Spectroscopy Reviews, 2021, 56, 733-763
work page 2021
-
[8]
C. Chen, Y . Zuo, W. Ye, X. Li, Z. Deng and S. P. Ong, Advanced Energy Materials, 2020, 10
work page 2020
Show all 58 references
-
[9]
Fedik, R
N. Fedik, R. Zubatyuk, M. Kulichenko, N. Lubbers, J. S. Smith, B. Nebgen, R. Messerly, Y . W. Li, A. I. Boldyrev, K. Barros, O. Isayev and S. Tretiak, Nature Reviews Chemistry, 2022, 6, 653-672
2022
-
[10]
Meuwly, Chemical Reviews, 2021, 121, 10218-10239
M. Meuwly, Chemical Reviews, 2021, 121, 10218-10239
2021
-
[11]
K. Chen, C. Guo, C. Wang, S. Zhao, B. Xiong, G. Lu, J. R. Reinfelder and Z. Dang, Water Research, 2024, 256, 121580
2024
-
[12]
Kanahashi, M
K. Kanahashi, M. Urushihara and K. Yamaguchi, Scientific Reports, 2022, 12, 11159
2022
-
[13]
C. Li, E. O. Adeniyi and P. Zarzycki, Scientific Reports, 2024, 14, 6603
2024
-
[14]
Scharrer, G
M. Scharrer, G. A. Agbanga, B. L. Brugman, M. Guild, G. Oliveira, J. Liu, X. Guo and A. Navrotsky, Materials Today, 2025, DOI: https://doi.org/10.1016/j.mattod.2025.05.010
2025 doi
- [15]
-
[16]
J. Liu, Z. Zhang, X. Li, M. Zong, Y . Wang, S. Wang, P. Chen, Z. Wan, L. Liu, Y . Liang, W. Wang, S. Wang, X. Guo, E. G. Saldanha, K. M. Rosso and X. Zhang, Chemical Engineering Journal, 2023, 473, 145216
2023
-
[17]
Zheng, O
Z. Zheng, O. Zhang, C. Borgs, J. T. Chayes and O. M. Yaghi, Journal of the American Chemical Society, 2023, 145, 18048-18062
2023
-
[18]
Dagdelen, A
J. Dagdelen, A. Dunn, S. Lee, N. Walker, A. S. Rosen, G. Ceder, K. A. Persson and A. Jain, Nature Communications, 2024, 15, 1418
2024
-
[19]
M. P. Polak and D. Morgan, Nature Communications, 2024, 15, 1569
2024
- [20]
-
[21]
Mistral OCR, https://mistral.ai/news/mistral-ocr, (accessed 05/21/2025, 2025)
2025
-
[22]
BerriAI, Journal, 2025
2025
- [23]
-
[24]
C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. Del Rio, M. Wiebe, P. Peterson, P. Gerard- Marchant, K. Sheppard, T...
2020
-
[25]
Reback, W
J. Reback, W. McKinney, jbrockmendel, J. V . d. Bossche, T. Augspurger, P. Cloud, gfyoung, S. Hawkins, Sinhrks, M. Roeschke, A. Klein, T. Petersen, J. Tratne, C. She, W. Ayd, S. Naveh, M. Garcia, J. Schendel, patrick, A. Hayden, D. Saxton, V . Jancauskas, A. McMaster, M. Gorel...
2021 doi
-
[26]
Thermochemical Database, https://theguolab.us/Thermochemical-database/index.html, (accessed 05/25/2025, 2025)
2025
-
[27]
Mentel, Journal, 2024
L. Mentel, Journal, 2024
2024
-
[28]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss and V . Dubourg, the Journal of machine Learning research, 2011, 12, 2825-2830
2011
-
[29]
N. L. Manakov, A. V . Meremianin and A. F. Starace, Physical Review A, 1998, 57, 3233- 3244
1998
- [30]
-
[31]
Akiba, S
T. Akiba, S. Sano, T. Yanase, T. Ohta and M. Koyama, 2019
2019
- [32]
- [33]
-
[34]
SC-Database, https://equilibriumdata.github.io/sc-database.html, (accessed 05/23/2025)
2025
-
[35]
Lewis, E
P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel and D. Kiela, presented in part at the Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 2020
2020
-
[36]
Bratsch and H
S. Bratsch and H. B. Silber, Polyhedron, 1982, 1, 219-223
1982
-
[37]
K. A. Gschneidner, Journal of the Less Common Metals, 1969, 17, 1-12
1969
- [38]
-
[39]
Reynolds and K
L. Reynolds and K. McDonell, presented in part at the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 2021
2021
-
[40]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin, Advances in neural information processing systems, 2017, 30
2017
-
[41]
F. B. Baker and C. E. Holley, Journal of Chemical & Engineering Data, 1968, 13, 405- 407
1968
-
[42]
Dash and N
S. Dash and N. K. Shukla, Journal of Thermal Analysis and Calorimetry, 2013, 112, 193- 200
2013
-
[43]
S. A. Gramsch and L. R. Morss, The Journal of Chemical Thermodynamics, 1995, 27, 551-560
1995
-
[44]
L. A. Reznitskii, Inorganic Materials, 2001, 37, 491-495
2001
-
[45]
N. D. Stout, C. L. Hoenig and P. C. Nordine, Journal of the American Ceramic Society, 1969, 52, 145-151
1969
-
[46]
S. W. Kieffer, Reviews of Geophysics, 1982, 20, 827-849
1982
-
[47]
S. K. Papadopoulou, A. de Souza Braga Neto, I. Billard, C. Cousin, V . Briois, A. Beauvois, L. Michot, G. Mériguet, A.-L. Rollet and J. Sirieix-Plénet, Separation and Purification Technology, 2025, 355, 129686
2025
-
[48]
J. Wei, X. Chu, X.-Y . Sun, K. Xu, H.-X. Deng, J. Chen, Z. Wei and M. Lei, InfoMat, 2019, 1, 338-358
2019
-
[49]
M. Ge, F. Su, Z. Zhao and D. Su, Materials Today Nano, 2020, 11, 100087
2020
-
[50]
Oviedo, J
F. Oviedo, J. L. Ferres, T. Buonassisi and K. T. Butler, Accounts of Materials Research, 2022, 3, 597-607
2022
-
[51]
A. S. Anker, K. T. Butler, R. Selvan and K. M. O. Jensen, Chem Sci, 2023, 14, 14003- 14019
2023
-
[52]
Bender, N
A. Bender, N. Schneider, M. Segler, W. Patrick Walters, O. Engkvist and T. Rodrigues, Nature Reviews Chemistry, 2022, 6, 428-442
2022
-
[53]
Debus, H
B. Debus, H. Parastar, P. Harrington and D. Kirsanov, TrAC Trends in Analytical Chemistry, 2021, 145, 116459
2021
-
[54]
Houhou and T
R. Houhou and T. Bocklitz, Analytical Science Advances, 2021, 2, 128-141
2021
-
[55]
J. Liu, X. Zhao, K. Zhao, V . G. Goncharov, J. Delhommelle, J. Lin and X. Guo, Scientific Reports, 2023, 13, 5919
2023
-
[56]
X. Zhao, Y . Luo, J. Liu, W. Liu, K. M. Rosso, X. Guo, T. Geng, A. Li and X. Zhang, The Journal of Physical Chemistry C, 2023, 127, 14830-14838
2023
-
[57]
C. J. Taylor, A. Pomberger, K. C. Felton, R. Grainger, M. Barecka, T. W. Chamberlain, R. A. Bourne, C. N. Johnson and A. A. Lapkin, Chemical Reviews, 2023, 123, 3089-3126
2023
-
[58]
Ahmadi, M
M. Ahmadi, M. Ziatdinov, Y . Zhou, E. A. Lass and S. V . Kalinin, Joule, 2021, 5, 2797- 2822
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.