REVIEW 3 major objections 5 minor 1 cited by
Thermodynamic assessment of machine learning models for solid-state synthesis prediction
T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Three of four ML synthesis models overpredict what can be made
desk verdict Useful benchmark paper: four synthesizability classifiers compared against thermodynamic heuristics on 2673 hypothetical oxides, with the finding that only SynthNN tracks the thermodynamic bounds; the temperature asymmetry between calibration and evaluation sets is a real caveat but does not overturn the main conclusion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the pair of thermodynamic bounds: Ehull (the energy of a material above the equilibrium phase-separation hull) and Γopt (the minimum, over all two-precursor reaction pathways, of a weighted sum of reaction energy and two competition terms). The paper recalibrates the weights on observed reactions, then uses 100 meV/atom as the cutoff for both quantities because 92% of successful reactions lie below Γopt = 100 meV/atom and roughly 95% of known materials lie below Ehull = 67 meV/atom. By computing these quantities for generated materials and binning model scores against them, the audit separates models that track thermodynamics from those that don't.
What would settle it
Run automated or human experimental synthesis attempts on a random sample of the 2673 generated materials across all four score regimes; if a substantial fraction of materials with Ehull and Γopt above 100 meV/atom are successfully made, the thermodynamic bounds are wrong and the claimed overprediction is an artifact of the heuristics. A cheaper version: check whether the models' scores separate materials from a curated database of failed synthesis attempts, which the paper identifies as the missing negative data.
Extended reading notes
Core claim
The paper establishes that most current ML synthesizability classifiers, despite high recall on test sets, systematically overpredict synthesizability when checked against thermodynamic heuristics, while SynthNN—a composition-based model trained with probabilistic labels on broadly sampled artificial formulas—produces scores that decline sharply for materials above the 100 meV/atom bounds on hull distance and reaction selectivity. It also validates Γopt, the most selective reaction pathway to a target, as a proxy for experimentally observed recipe selectivity: 41% of text-mined reactions use exactly the optimal precursors and 63% are within 20 meV/atom of optimum.
Load-bearing premise
Thermodynamic plausibility, measured through CHGNet energies and a two-precursor reaction model, accurately identifies which materials are truly unlikely to be synthesized; if the enthalpy reference is biased or important reaction routes are missing, the bounds change and the overprediction verdict could shift.
Editorial extensions
If this is right
- Screening pipelines that rely on PU-CGNF, PU-CGCNN, or SynCoTrain will pass many candidates with Ehull and Γopt beyond the range of known successful reactions, wasting experimental effort.
- SynthNN's training recipe—artificial unlabeled formulas plus confidence-weighted labels—produces scores aligned with thermodynamics and is a template for better synthesis predictors.
- Γopt adds independent information beyond Ehull: among materials with Ehull < 100 meV/atom, about 60% still have Γopt > 100 meV/atom, so selectivity screening can cut the candidate pool by more than half.
- Model disagreement is large (only 7% unanimous predictions), so ensembling current models would not produce a reliable synthesizability filter.
- Directly incorporating thermodynamic quantities into training and validation is a promising route to models whose scores can be trusted as probabilities.
Reading between the lines
- The audit's thermodynamic bounds are only as good as the successful-reaction literature; reported syntheses are a biased sample, so the 100 meV/atom cutoffs may be stricter in practice—reported successes skew toward easy targets.
- The same audit could be applied to other synthesis routes (solution, mechanochemical) where different thermodynamic/kinetic balances apply, potentially shifting bounds and model rankings.
- If CHGNet systematically softens Ehull for generated materials, then the fraction of 'thermodynamically plausible' candidates is an upper bound, and the overprediction the paper reports is likely a lower bound on the true gap.
- The near-zero correlation among model scores suggests the models are learning different heuristics; a testable extension is to combine each model's score with Ehull/Γopt in a single funnel to see which ordering best predicts actual synthesis outcomes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper calibrates a thermodynamic selectivity metric Γ (reaction energy plus primary and secondary competition terms) on a text-mined dataset of successful solid-state synthesis reactions, proposes heuristic feasibility bounds (Ehull < 100 meV/atom and Γopt < 100 meV/atom), computes these quantities with the CHGNet machine-learned potential for 2673 Chemeleon-generated ternary oxide hypotheticals, and evaluates four positive-unlabeled synthesizability classifiers (PU-CGNF, SynthNN, PU-CGCNN, SynCoTrain) against these thermodynamic references. The central claim is that most current synthesizability models overpredict the likelihood of synthesis, with SynthNN being the only model whose scores decrease strongly with worsening Ehull and Γopt.
Significance. If the conclusions hold, the paper provides a timely, quantitative benchmark for an emerging class of synthesizability predictors. It demonstrates that three of four widely used models are overoptimistic filters for hypothetical materials and identifies training-data and labeling choices that correlate with thermodynamic alignment. The authors' practice of releasing the workflow and data, evaluating multiple independently trained models on a common dataset, and explicitly acknowledging the limitations of CHGNet energy errors and the two-precursor/no-byproduct restriction adds credibility. The comparison is meaningful even though the thermodynamic heuristics themselves are approximate, because the paper is explicit about what the heuristics can and cannot capture.
major comments (3)
- [Methods, p.18; Figure 2; Table 2] There is a temperature-protocol asymmetry between the calibration and evaluation sets. For text-mined targets, Γobs and Γopt are evaluated at the reported synthesis temperature ('thermodynamics ... updated based on the synthesis temperature of interest'). For Chemeleon-generated materials, Γopt is minimized over 600, 900, 1200, 1500, and 1800 K ('chose the optimum reaction as that with the minimum Γ across all temperatures'). Since Γ varies with temperature, this makes the evaluation-set Γopt systematically lower than the calibration-set Γopt used to define the 100 meV/atom threshold. Consequently, the 16% figure in Table 2 and the x-axis in Figure 5 are computed under a different definition of Γopt than the bound itself. Please either compute Γopt for the generated materials under the same temperature protocol as the calibration data, or provide a quantitative sensitivity analysis showi
- [Methods, Eq. (2); Figure 2b] The weights w0, w1, w2 in Eq. (2) are fitted by minimizing the L2 difference between Γopt and Γobs on the same text-mined dataset whose Γobs distribution is then used to set the 100 meV/atom Γopt bound (92nd percentile of Γobs, 96th percentile of Γopt in Figure 2b). Γopt is therefore not an independent first-principles standard; the agreement between Γobs and Γopt shown in Figure 2a is partly achieved by construction. This does not invalidate the heuristic, but the threshold should be validated with held-out reactions or with un-fitted weights to establish that the 100 meV/atom cutoff is not an artifact of calibration.
- [p.8; p.18; Figure 3] The paper notes that CHGNet slightly underestimates Ehull for generated materials (Refs. 34 and 46), while text-mined energies are MP GGA/GGA+U values. The magnitude of this bias is not propagated into the reported fractions (40%, 16%) or into the moving-average trends in Figures 4 and 5. If the bias is tens of meV/atom and composition-dependent, the fraction of generated materials within the thermodynamic bounds, and possibly the relative ordering of models, could shift. Please quantify the sensitivity of the main conclusions to this energy offset, for example by recomputing a subset with DFT or by shifting the distributions by the known error, and report how the central claims change.
minor comments (5)
- [Abstract / Discussion] The abstract states that four ML models are assessed, but the Discussion and Figure S3 also evaluate TSDNN. Please either mention the fifth model in the abstract or explicitly frame it as an additional exploratory check.
- [Figure 1] The caption says the examples in 1(b) are indicated with red and blue points, but the points in 1(a) are grey. Please make the marker colors consistent and legible.
- [p.7] 'Allowing for some outliers, materials with a Ehull > 100 meV/atom are similarly considered unlikely to be synthesizable' is vague. Since the 100 meV/atom Ehull bound is central, please cite a quantitative percentile justification or state explicitly that it is an ad hoc extension of Sun et al.'s 67 meV/atom result.
- [Figure S3] Typo: 'Chemelon-generated materials' should be 'Chemeleon-generated materials'.
- [Figures 4 and 5] The moving averages of the fraction predicted synthesizable are informative, but confidence intervals or the number of materials per bin would help assess whether the differences between models are significant, particularly in the sparsely populated high-Ehull and high-Γopt regimes.
Circularity Check
Γopt is calibrated on the same text-mined reactions used to validate it, but the central ML-versus-thermodynamics comparison is otherwise independent.
-
fitted input called prediction
[Methods (Calculating thermodynamic selectivity, Eq. 2) and Results (Calibration on text-mined synthesis data, Fig. 2a)]
"Under the assumption that the optimum (lowest) Γ should correspond with the most likely reaction, we re-optimized the weights w0, w1, and w2 by minimizing the L2 loss between the lowest Γ (Γopt) and the Γ associated with the observed reaction (Γobs)... The highest density of points lies near the parity line, indicating that observed (successful) reactions often approach the optimal selectivity."
The weights in Eq. 2 are fit by minimizing the L2 difference between Γopt and Γobs on the same text-mined reactions, so the near-parity between Γopt and Γobs in Fig. 2a is partly enforced by construction and cannot serve as independent validation of Γopt as a proxy for Γobs. The 100 meV/atom Γopt bound and the '16% of materials fall within both proposed thermodynamic bounds' figure inherit this fitted relationship. This is a real but partial circularity: it affects the Γ-based heuristic itself, not the external ML predictions.
full rationale
The paper's central comparison—four pretrained ML synthesizability models versus thermodynamic heuristics on 2673 Chemeleon-generated materials—is largely independent: the ML scores come from external pretrained models, Ehull is computed with CHGNet/MP energetics, and the generated materials are new to all models. The overprediction conclusion is also supported by the Ehull-only bound (40% vs 50–77% predicted synthesizable), which does not depend on the fitted Γ weights. The main circular element is internal: Γopt is calibrated by fitting its weights to text-mined Γobs values and then the same dataset is used to claim Γopt is a good proxy and to set the Γopt < 100 meV/atom threshold. The paper is transparent about this calibration, describing Γ as a heuristic rather than a first-principles standard. Self-citations (Refs. 34, 46) are used only to caveat CHGNet's softening, not as load-bearing uniqueness claims. The temperature asymmetry noted by the skeptic—Γopt minimized over 600–1800 K for generated materials but evaluated at observed synthesis temperatures for the calibration set—is a methodological bias risk, not circularity, and would if anything make the thermodynamic bound more permissive for the generated set. Overall, the derivation is not an equation-equals-equation circularity, and the core ML-vs-thermodynamics finding retains independent content, warranting a low-to-moderate circularity score.
Assumptions & free parameters
free parameters (3)
- Γ weights (w0, w1, w2) =
0.16, 0.56, 0.28
- Thermodynamic feasibility bounds =
Ehull < 100 meV/atom; Γopt < 100 meV/atom
- Synthesis temperature grid for Γopt =
600, 900, 1200, 1500, 1800 K
assumptions (7)
- domain assumption MP DFT GGA/GGA+U energies are accurate ground-state energies for all known phases in the chemical spaces.
- domain assumption CHGNet v0.3.0 total energies can be mixed with MP DFT energies to compute Ehull and Γopt of hypothetical materials.
- domain assumption The Bartel et al. Gibbs-energy model extrapolates 0 K DFT energies to synthesis temperatures accurately.
- domain assumption Two-precursor reactions with no solid byproducts suffice to capture realistic synthesis selectivity.
- domain assumption The Kononova text-mined recipes are successful syntheses with correct target/precursor/temperature extraction.
- domain assumption PU-learning model scores are comparable probability-like outputs across models.
- domain assumption Pretrained synthesizability models generalize to Chemeleon-generated materials outside their training distributions.
Cite this review
Pith. "Pith review of Thermodynamic assessment of machine learning models for solid-state synthesis prediction." pith.science (2026). https://pith.science/paper/G4XY5GKI
@misc{pith2026260204075,
author = {Pith},
title = {Pith review of: Thermodynamic assessment of machine learning models for solid-state synthesis prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/G4XY5GKI}},
note = {Machine review of arXiv:2602.04075}
}
read the original abstract
Machine learning models have recently emerged to predict whether hypothetical solid-state materials can be synthesized. These models aim to circumvent direct first-principles modeling of solid-state phase transformations, instead learning from large databases of successfully synthesized materials. Here, we assess the alignment of several recently introduced synthesis prediction models with material and reaction thermodynamics, quantified by the energy with respect to the convex hull and a metric accounting for thermodynamic selectivity of enumerated synthesis reactions. A dataset of successful synthesis recipes was used to determine the likely bounds on both quantities beyond which materials can be deemed unlikely to be synthesized. With these bounds as context, thermodynamic quantities were computed using the CHGNet foundation potential for thousands of new hypothetical materials generated using the Chemeleon generative model. Four recently published machine learning models for synthesizability prediction were applied to this same dataset, and the resultant predictions were considered against computed thermodynamics. We find these models generally overpredict the likelihood of synthesis, but some model scores do trend with thermodynamic heuristics, assigning lower scores to materials that are less stable or do not have an available synthesis recipe that is calculated to be thermodynamically selective. In total, this work identifies existing gaps in machine learning models for materials synthesis and introduces a new approach to assess their quality in the absence of extensive negative examples (failed syntheses).
Forward citations
Cited by 1 Pith paper
-
Rapid estimation of synthesizability windows of inorganic materials from first principles
A hybrid DFT and machine-learned potential approach produces predominance diagrams for binary and 48 ternary metal phosphosulfide compounds, showing agreement with experiments and identifying synthesis windows where s...
Reference graph
Works this paper leans on
-
[2]
Number and fraction of Chemeleon-generated hypothetical materials predicted to be synthesizable by four ML classifiers, along with subsets of materials that fall within realistic thermodynamic bounds. # of materials % of materials Total hypothetical materials considered 2673 N/A Predicted synthesizable (PU-CGNF) 2050 77% Predicted synthesizable (SynthNN) ...
-
[3]
#$+0.56 𝐶%+0.28 𝐶& (2) 18 For novel materials with unknown synthesis recipes, we generated reactions at 600, 900, 1200, 1500, and 1800 K and chose the “optimum
Γ=0.16 ∆𝐺"#$+0.56 𝐶%+0.28 𝐶& (2) 18 For novel materials with unknown synthesis recipes, we generated reactions at 600, 900, 1200, 1500, and 1800 K and chose the “optimum” reaction as that with the minimum Γ across all temperatures. Novel materials generation Novel materials were generated using the generative model Chemeleon as detailed at https://github....
2016
-
[7]
(13) Neumann, M.; Gin, J.; Rhodes, B.; Bennett, S.; Li, Z.; Choubisa, H.; Hussey, A.; Godwin, J
https://doi.org/10.48550/ARXIV .2506.23971. (13) Neumann, M.; Gin, J.; Rhodes, B.; Bennett, S.; Li, Z.; Choubisa, H.; Hussey, A.; Godwin, J. Orb: A Fast, Scalable Neural Network Potential. arXiv
-
[9]
https://doi.org/10.48550/ARXIV .2405.04967. (15) Bartel, C. J. Review of Computational Approaches to Predict the Thermodynamic Stability of Inorganic Solids. J. Mater. Sci. 2022, 57 (23), 10475–10498. https://doi.org/10.1007/s10853-022-06915-4. (16) Zagorac, D.; Müller, H.; Ruehl, S.; Zagorac, J.; Rehme, S. Recent Developments in the Inorganic Crystal Str...
-
[19]
https://doi.org/10.1038/s41524-024-01500-6. (47) Ong, S. P.; Richards, W. D.; Jain, A.; Hautier, G.; Kocher, M.; Cholia, S.; Gunter, D.; Chevrier, V . L.; Persson, K. A.; Ceder, G. Python Materials Genomics (Pymatgen): A Robust, Open-Source Python Library for Materials Analysis. Comput. Mater. Sci. 2013, 68, 314–319. https://doi.org/10.1016/j.commatsci.20...
-
[97]
(11) Deng, B.; Zhong, P.; Jun, K.; Riebesell, J.; Han, K.; Bartel, C
https://doi.org/10.1038/s41524-020-00362-y. (11) Deng, B.; Zhong, P.; Jun, K.; Riebesell, J.; Han, K.; Bartel, C. J.; Ceder, G. CHGNet as a Pretrained Universal Neural Network Potential for Charge-Informed Atomistic Modelling. Nat. Mach. Intell. 2023, 5 (9), 1031–1041. https://doi.org/10.1038/s42256-023-00716-3. (12) Wood, B. M.; Dzamba, M.; Fu, X.; Gao, ...
-
[121]
https://doi.org/10.1038/s41570-018-0121. (4) Horton, M. K.; Huck, P.; Yang, R. X.; Munro, J. M.; Dwaraknath, S.; Ganose, A. M.; Kingsbury, R. S.; Wen, M.; Shen, J. X.; Mathis, T. S.; Kaplan, A. D.; Berket, K.; Riebesell, J.; George, J.; Rosen, A. S.; Spotte-Smith, E. W. C.; McDermott, M. J.; Cohen, O. A.; Dunn, A.; Kuner, M. C.; Rignanese, G.-M.; Petretto...
-
[155]
(31) Amariamir, S.; George, J.; Benner, P
https://doi.org/10.1038/s41524-023-01114-4. (31) Amariamir, S.; George, J.; Benner, P. SynCoTrain: A Dual Classifier PU-Learning Framework for Synthesizability Prediction. Digit. Discov. 2025, 4 (6), 1437–1448. https://doi.org/10.1039/D4DD00394B. (32) Gleaves, D.; Fu, N.; Dilanga Siriwardane, E. M.; Zhao, Y .; Hu, J. Materials Synthesizability and Stabili...
Show all 21 references
-
[185]
(52) Schütt, K
https://doi.org/10.1038/s41524-021-00650-1. (52) Schütt, K. T.; Kessel, P.; Gastegger, M.; Nicoli, K. A.; Tkatchenko, A.; Müller, K.-R. SchNetPack: A Deep Learning Toolbox For Atomistic Systems. J. Chem. Theory Comput. 2019, 15 (1), 448–455. https://doi.org/10.1021/acs.jctc.8b...
2019 doi
-
[203]
(27) Chung, V .; Walsh, A.; J
https://doi.org/10.1038/s41597-019-0224-1. (27) Chung, V .; Walsh, A.; J. Payne, D. Solid-State Synthesizability Predictions Using Positive-Unlabeled Learning from Human-Curated Literature Data. Digit. Discov. 2025, 4 (9), 2439–2453. https://doi.org/10.1039/D5DD00065C. (28) Ja...
2025 doi
-
[231]
(39) Wang, Z.; Sun, Y .; Cruse, K.; Zeng, Y .; Fei, Y .; Liu, Z.; Shangguan, J.; Byeon, Y .-W.; Jun, K.; He, T.; Sun, W.; Ceder, G
https://doi.org/10.1038/s41597-022-01317-2. (39) Wang, Z.; Sun, Y .; Cruse, K.; Zeng, Y .; Fei, Y .; Liu, Z.; Shangguan, J.; Byeon, Y .-W.; Jun, K.; He, T.; Sun, W.; Ceder, G. Optimal Thermodynamic Conditions to Minimize Kinetic By-Products in Aqueous Materials Synthesis. Nat....
2024 doi
-
[1998]
E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; van der Walt, S
(44) Virtanen, P.; Gommers, R.; Oliphant, T. E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; van der Walt, S. J.; Brett, M.; Wilson, J.; Millman, K. J.; Mayorov, N.; Nelson, A. R. J.; Jones, E.; Kern, R.; Larson, E.; Carey, ...
2020
-
[2018]
(22) Szymanski, N
https://doi.org/10.1038/s41467-019-10030-5. (22) Szymanski, N. J.; Nevatia, P.; Bartel, C. J.; Zeng, Y .; Ceder, G. Autonomous and Dynamic Precursor Selection for Solid-State Materials Synthesis. Nat. Commun. 2023, 14 (1),
2023 doi
- [2024]
-
[2025]
(5) Park, H.; Li, Z.; Walsh, A
https://doi.org/10.1038/s41563-025-02272-0. (5) Park, H.; Li, Z.; Walsh, A. Has Generative Artificial Intelligence Solved Inverse Materials Design? Matter 2024, 7 (7), 2355–2367. https://doi.org/10.1016/j.matt.2024.05.017. (6) Jain, A.; Ong, S. P.; Hautier, G.; Chen, W.; Richa...
2024 doi
-
[3097]
(25) McDermott, M
https://doi.org/10.1038/s41467-021-23339-x. (25) McDermott, M. J.; McBride, B. C.; Regier, C. E.; Tran, G. T.; Chen, Y .; Corrao, A. A.; Gallant, M. C.; Kamm, G. E.; Bartel, C. J.; Chapman, K. W.; Khalifah, P. G.; Ceder, G.; Neilson, J. R.; Persson, K. A. Assessing Thermodynam...
2023 doi
-
[4168]
(43) Chase, M
https://doi.org/10.1038/s41467-018-06682-4. (43) Chase, M. NIST-JANAF Thermochemical Tables, 4th Edition; American Institute of Physics, -1,
-
[4379]
(34) Szymanski, N
https://doi.org/10.1038/s41467-025-59636-y. (34) Szymanski, N. J.; Bartel, C. J. Establishing Baselines for Generative Discovery of Inorganic Crystals. Mater. Horiz. 2025, 12 (19), 8000–8011. https://doi.org/10.1039/D5MH00010F. (35) Riebesell, J.; Goodall, R. E. A.; Benner, P....
2025 doi
-
[5388]
(9) Calderon, C
https://doi.org/10.21105/joss.05388. (9) Calderon, C. E.; Plata, J. J.; Toher, C.; Oses, C.; Levy, O.; Fornari, M.; Natan, A.; Mehl, M. J.; Hart, G.; Buongiorno Nardelli, M.; Curtarolo, S. The AFLOW Standard for High- 21 Throughput Materials Science Calculations. Comput. Mater...
2015 doi
-
[6280]
(49) Zhou, Q.; Tang, P.; Liu, S.; Pan, J.; Yan, Q.; Zhang, S.-C
https://doi.org/10.1038/s41467-020-19964-7. (49) Zhou, Q.; Tang, P.; Liu, S.; Pan, J.; Yan, Q.; Zhang, S.-C. Learning Atoms for Materials Discovery. Proc. Natl. Acad. Sci. 2018, 115 (28). https://doi.org/10.1073/pnas.1801181115. (50) Xie, T.; Grossman, J. C. Crystal Graph Conv...
2018 doi
-
[6956]
(23) Szymanski, N
https://doi.org/10.1038/s41467-023-42329-9. (23) Szymanski, N. J.; Byeon, Y .-W.; Sun, Y .; Zeng, Y .; Bai, J.; Kunz, M.; Kim, D.-M.; Helms, B. A.; Bartel, C. J.; Kim, H.; Ceder, G. Quantifying the Regime of Thermodynamic Control 22 for Solid-State Reactions during Ternary Met...
2024 doi
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.