Pith. sign in

REVIEW 4 major objections 4 minor 20 references

A Benchmark for Quantum Chemistry Relaxations via Machine Learning Interatomic Potentials

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper turns 7 TB of raw PubChemQC geometry-optimization logs into PubChemQCR, a benchmark of about 3.5 million relaxation trajectories and 300 million labeled conformations, and benchmarks nine interatomic-potential models on it.

desk verdict A genuinely useful large-scale relaxation-trajectory dataset, but the missing label validation and a few overclaims need fixing before I'd trust the numbers. read the letter →

arxiv 2506.23008 v2 pith:6WMSLKPC submitted 2025-06-28 q-bio.QM

Cong Fu , Yuchao Lin , Zachary Krueger , Wendi Yu , Xiaoning Qian , Byung-Jun Yoon , Raymundo Arróyave , Xiaofeng Qian
show 3 more authors
Toshiyuki Maeda Maho Nakata Shuiwang Ji
This is my paper
classification q-bio.QM
keywords machinelearninginteratomicpotentialsquantumchemistrydatasetDFTrelaxationtrajectoriesenergyandforcelabelsPubChemQCRmolecularconformationsgeometryoptimizationbenchmarksmallorganicmolecules
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that machine-learned interatomic potentials need training and evaluation data that include the off-equilibrium geometries encountered during relaxation, not just final stable structures, and it builds the missing resource at scale. The authors curate the raw geometry-optimization logs of the PubChemQC project into PubChemQCR, which they state is the largest publicly available dataset of DFT-based relaxation trajectories for small organic molecules, with roughly 3.5 million trajectories and over 300 million snapshots, each labeled with total energy and atomic forces. They also provide baseline results for nine machine-learning interatomic potentials on a smaller subset and for two of those models on the full dataset, together with tools for loading the data. A sympathetic reader would take away that trajectory-level supervision is now available in a form that can be used to train and compare MLIPs directly.

What carries the argument

The carrying object is PubChemQCR itself, a dataset of relaxation trajectories built by parsing raw PubChemQC log files into memory-mapped database records keyed by PubChem compound ID. Its structure matters: each trajectory is a sequence of snapshots containing atomic numbers, Cartesian coordinates, total energy, and force gradients, organized by the four optimization stages, so a model can be trained on just the DFT-stage portion where labels are most accurate. The paper also computes isolated atomic energies at B3LYP/6-31G* and subtracts them to form energy targets, which centers the energy distribution and makes the regression task easier.

What would settle it

Recompute total energies and atomic forces with B3LYP/6-31G* for a random sample of a few hundred parsed snapshots (or check that stored forces equal negative gradients of stored energies by finite differences); a systematic mismatch in energy offsets, force magnitudes, or signs would falsify the curation claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a careful parsing of the 7 TB of PubChemQC optimization outputs yields a reusable benchmark: approximately 3.5 million relaxation trajectories and 298.75 million molecular snapshots, of which 105.49 million come from the DFT stage, spanning 25 chemical elements. Energy and force labels are stored per snapshot, grouped by optimization stage (PM3, Hartree–Fock, DFT via Firefly/SMASH, and DFT via GAMESS), and formation energies are defined by subtracting isolated atomic energies so the learning target is compactly distributed. On the benchmark, Equiformer achieves the best energy and force errors on the 40,979-trajectory subset, and PaiNN performs best when trained on the full dataset; geometry-optimization experiments show that most models struggle to drive near-equilibrium structures to convergence, with Equiformer the clear outlier. The paper further claims the dataset is the largest of its kind and that its trajectory-level splits avoid information leakage.

Load-bearing premise

The dataset's validity rests on the assumption that parsing 7 TB of raw log files into atomic numbers, coordinates, total energy, and forces is error-free in units, signs, and stage assignment, and the paper reports no independent check of the parsed labels against recomputed DFT values.

Editorial extensions

If this is right

  • Training on PubChemQCR exposes MLIPs to intermediate, non-equilibrium conformations, not only relaxed minima, which is the regime molecular dynamics actually samples.
  • The benchmark numbers establish a reference point: Equiformer leads on the small subset, while PaiNN leads on the full dataset, so later models can be compared against these numbers.
  • Trajectory-level data splits mean a model is evaluated on molecules whose relaxation paths were unseen during training, testing generalization rather than memorization.
  • The formation-energy normalization removes per-atom offsets and is reported to speed convergence of the learned energy model.
  • Geometry optimization results quantify a remaining gap: from near-equilibrium starting points, only Equiformer reaches chemical accuracy on a substantial fraction of test molecules.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the labels survive independent validation, this corpus could serve as a supervised pretraining source for molecular property prediction, since it couples geometry with physical energy and force supervision.
  • A natural stress test is cross-stage training: train on DFT-stage labels and evaluate on PM3/HF snapshots; the paper's own limitation section predicts large inconsistency, and quantifying it would tell users how much of the multi-level data is usable.
  • The 25-element coverage and near-equilibrium force distribution mean the dataset is not by itself a substitute for active learning or MD sampling that must explore high-force regions; pairing it with on-the-fly DFT queries is the obvious next step.
  • Because forces are stored per snapshot, PubChemQCR could be used to test energy-conservation and equivariance properties of MLIP architectures on far more diverse molecules than in earlier MD datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces PubChemQCR, a dataset of molecular geometry optimization trajectories curated from roughly 7 TB of raw PubChemQC log files. It reports approximately 3.5 million trajectories and 298,751,667 snapshots across PM3, Hartree–Fock, and two DFT substages, with each snapshot labeled by total energy and atomic forces. The authors benchmark nine machine learning interatomic potentials on a smaller subset (PubChemQCR-S) and two models on the full dataset, and they evaluate geometry optimization performance. The central claim is that PubChemQCR is the largest publicly available dataset of DFT-based relaxation trajectories for small organic molecules and is suitable for training and evaluating MLIPs.

Significance. If the curation is correct, PubChemQCR is a valuable and reusable resource: it is large, publicly released with a custom PyTorch Geometric dataloader, organized into trajectory-level splits, and includes both energy and force labels. The benchmark provides a useful reference point, and the limitations section is candid about the near-equilibrium bias of the DFT-stage data. However, the manuscript's central value depends on the fidelity of the parsed labels, and the absence of any independent validation of energies, forces, and stage assignment is a significant correctness risk. There is also a concrete internal inconsistency in the headline snapshot count. These issues are fixable within the scope of the paper, but they need to be addressed before the resource can be relied upon.

major comments (4)
  1. [Section 3.3] The core value of PubChemQCR rests on parsing 7 TB of raw log files into atomic numbers, coordinates, energies, and forces, but no validation of these parsed labels is reported. A systematic error in units (Hartree vs eV), force sign (gradient vs force), or stage assignment (PM3/HF/DFT_1st/DFT_2nd) would silently propagate into every baseline in Tables 3–5. Please report quantitative checks: for example, recompute a random subset of snapshots with an independent DFT code and compare energies and forces, verify force directions and magnitudes at stationary points, and confirm that atomic numbers and stage labels match the original PubChemQC records. This validation is load-bearing for the central claim that the dataset provides high-quality energy and force labels.
  2. [Table 1 / Abstract / Section 7] Table 1 sums to 298,751,667 snapshots, yet the abstract and Section 7 claim 'over 300 million conformations.' This is an internal numerical inconsistency in a headline quantity. The text should be changed to 'approximately 298.8 million' or the counts should be revised so that the reported total actually exceeds 300 million.
  3. [Section 4.1 / Table 4] The full-dataset benchmarks are said to use 'data from the DFT optimization stage,' which in Section 3.4 includes both DFT_1st (Firefly/SMASH) and DFT_2nd (GAMESS) substages. Section 6 explicitly states that labels from different stages of the same trajectory are 'neither directly comparable nor mutually consistent.' If Table 4 pools DFT_1st and DFT_2nd labels, the training targets are heterogeneous; please either report results separately for each substage or provide a clear justification for pooling them. This affects the interpretation of the full-dataset results.
  4. [Tables 3–5] The benchmark tables report single runs without standard deviations over training seeds. Given the small gaps in Table 3 (e.g., test energy MAE of 5.38 meV/atom for Equiformer versus 5.33 for PaiNN), seed variability could change the reported rankings. Please report mean and standard deviation over at least three seeds for the main energy/force and geometry optimization metrics, or otherwise demonstrate that the differences are statistically meaningful.
minor comments (4)
  1. [Section 4.1] The text contains a typo: 'NeuqIP' should be 'NequIP' in the Results and Discussions paragraph.
  2. [Section 4.1] The learning-rate scheduler is described as 'REDUCE LRO NPLATEAU'; this should be formatted as 'ReduceLROnPlateau' for readability.
  3. [Section 2] The 'largest publicly available dataset' claim would be easier to verify if the paper gave a direct quantitative comparison with the OMol25 trajectory counts under the same scope definition (small organic molecules), rather than listing OMol25 properties only descriptively.
  4. [Section 6] The limitation about near-equilibrium forces is candid and important; it would be helpful to add a short discussion of how future extensions might include off-equilibrium sampling, since this is the main regime where MLIPs are expected to be used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; PubChemQCR is an empirical resource whose benchmarks are externally evaluated, and the self-cited prior datasets are data sources rather than load-bearing derivations.

full rationale

PubChemQCR does not derive a theoretical result from its inputs. Its central claim is the existence and scale of a curated dataset parsed from PubChemQC raw logs; the curation pipeline (Sections 3.2-3.3) transforms external quantum-chemistry output into LMDB records, with no equation in which a predicted quantity is defined in terms of the benchmark target. The benchmark evaluations (Tables 3-5) compare trained models against held-out DFT labels on disjoint trajectories, so they are standard empirical assessments rather than fitted-input predictions. The only same-group citations (PubChemQC, Molecule3D, PCQM4Mv2) are used to identify the source database and related datasets, not to justify a mathematical claim or to forbid alternative choices. The energy normalization in Section 3.3 defines formation energy as total energy minus per-atom constants; this is preprocessing, not a prediction. The unvalidated label parsing and the 298,751,667 vs 'over 300 million' count discrepancy are correctness and data-quality issues, not circularity, and should be reported under data-quality risk rather than raising the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced; the dataset is a derived artifact from existing PubChemQC calculations. The load-bearing assumptions are the correctness of the original calculations, the correctness of the parsing pipeline, the consistency of the two DFT codes used, and the adequacy of near-equilibrium sampling for MLIP training.

free parameters (2)
  • Baseline model hyperparameters = Table 2; Equiformer reduced to 4 layers with irreps 128x0e+64x1e
    Model rankings in Tables 3-5 depend on hand-chosen layer counts, hidden dimensions, batch sizes, and the reduced Equiformer configuration adopted because the original OC20 settings failed to converge; these choices are not swept and could affect conclusions.
  • Interaction cutoff radius = 4.5 Angstrom
    Chosen uniformly for all benchmark models in Section 4.1; changes the local atomic environment each model sees and can alter accuracy for molecules with longer-range interactions.
assumptions (4)
  • domain assumption PubChemQC raw geometry optimization calculations are accurate and complete at the B3LYP/6-31G* level.
    Section 3.2 relies on Firefly, SMASH, and GAMESS outputs from PubChemQC without independent re-validation of the original electronic structure calculations.
  • domain assumption Parsing the raw log files into energy, force, and coordinate tuples is correct.
    Section 3.3 describes removing failed and duplicate logs but reports no quality check on a random subset of parsed labels; a systematic parsing error would affect all downstream results.
  • domain assumption DFT_1st and DFT_2nd snapshots are consistent enough to be pooled as a single DFT stage.
    Section 3.3 groups Firefly/SMASH and GAMESS snapshots under 'DFT', and Section 4.1 trains on 'the DFT optimization stage' without a code-to-code consistency analysis.
  • domain assumption The DFT-stage trajectories provide sufficient diversity for training generalizable MLIPs.
    Section 6 acknowledges the data are confined to near-equilibrium geometries with small forces, which limits force diversity and off-equilibrium coverage relative to the abstract's motivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Benchmark for Quantum Chemistry Relaxations via Machine Learning Interatomic Potentials." pith.science (2026). https://pith.science/paper/6WMSLKPC

@misc{pith2026250623008,
  author       = {Pith},
  title        = {Pith review of: A Benchmark for Quantum Chemistry Relaxations via Machine Learning Interatomic Potentials},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6WMSLKPC}},
  note         = {Machine review of arXiv:2506.23008}
}
read the original abstract

Computational quantum chemistry plays a critical role in drug discovery, chemical synthesis, and materials science. While first-principles methods, such as density functional theory (DFT), provide high accuracy in modeling electronic structures and predicting molecular properties, they are computationally expensive. Machine learning interatomic potentials (MLIPs) have emerged as promising surrogate models that aim to achieve DFT-level accuracy while enabling efficient large-scale atomistic simulations. The development of accurate and transferable MLIPs requires large-scale, high-quality datasets with both energy and force labels. Critically, MLIPs must generalize not only to stable geometries but also to intermediate, non-equilibrium conformations encountered during atomistic simulations. In this work, we introduce PubChemQCR, a large-scale dataset of molecular relaxation trajectories curated from the raw geometry optimization outputs of the PubChemQC project. PubChemQCR is the largest publicly available dataset of DFT-based relaxation trajectories for small organic molecules, comprising approximately 3.5 million trajectories and over 300 million molecular conformations computed at various levels of theory. Each conformation is labeled with both total energy and atomic forces, making the dataset suitable for training and evaluating MLIPs. To provide baselines for future developments, we benchmark nine representative MLIP models on the dataset. Our resources are publicly available at https://huggingface.co/divelab

Figures

Figures reproduced from arXiv: 2506.23008 by the authors.

Figure 1
Figure 1. An overview of the PubChemQCR dataset. We curate raw geometry optimization tra [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of energy from the DFT optimization stage. (a) Total energy. (b) Formation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Isolated atomic energy calculated at DFT B3LYP/6- [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Distribution of forces from the DFT optimization stage. The full PubChemQCR dataset comprises 3,471,000 molecular relaxation trajectories and a total of 298,751,667 molecular snapshots spanning multiple levels of quantum chemical theory. Specifically, as shown in [PIT…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 7 canonical work pages

  1. [5]

    The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts

    Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, et al. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts. ACS Catalysis, 13(5):3066–3084,

    work page 2022

  2. [6]

    Kuzma Khrabrov, Anton Ber, Artem Tsypin, Konstantin Ushenin, Egor Rumiantsev, Alexander Telepov, Dmitry Protasov, Ilya Shenbin, Anton Alekseev, Mikhail Shirokikh, et al

    doi: 10.1038/ s42256-023-00716-3. Kuzma Khrabrov, Anton Ber, Artem Tsypin, Konstantin Ushenin, Egor Rumiantsev, Alexander Telepov, Dmitry Protasov, Ilya Shenbin, Anton Alekseev, Mikhail Shirokikh, et al. ∇2 dft: A universal quantum chemistry dataset of drug-like molecules and a benchmark for neural network potentials. arXiv preprint arXiv:2406.14347,

    arXiv

  3. [7]

    The open molecules 2025 (omol25) dataset, evaluations, and models

    Daniel S Levine, Muhammed Shuaibi, Evan Walter Clark Spotte-Smith, Michael G Taylor, Muham- mad R Hasyim, Kyle Michel, Ilyes Batatia, Gábor Csányi, Misko Dzamba, Peter Eastman, et al. The open molecules 2025 (omol25) dataset, evaluations, and models. arXiv preprint arXiv:2505.08762,

    arXiv 2025

  4. [9]

    URL https://openreview.net/forum?id=7DAFwp0Vne

    ISSN 2835-8856. URL https://openreview.net/forum?id=7DAFwp0Vne. Featured Certification. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  5. [11]

    GeoMFormer: A General Architecture for Geometric Molecular Representation Learning

    Tianlang Chen, Shengjie Luo, Di He, Shuxin Zheng, Tie-Yan Liu, and Liwei Wang. Geom- former: A general architecture for geometric molecular representation learning. arXiv preprint arXiv:2406.16853,

    work page Pith review arXiv

  6. [13]

    Strategies for pre-training graph neural networks

    Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265,

    arXiv 1905

  7. [15]

    Sliced Denoising: A Physics-Informed Molecular Pre-Training Method

    Yuyan Ni, Shikun Feng, Wei-Ying Ma, Zhi-Ming Ma, and Yanyan Lan. Sliced denoising: A physics-informed molecular pre-training method. arXiv preprint arXiv:2311.02124,

    work page Pith review arXiv

  8. [16]

    Generalizing denoising to non-equilibrium structures improves equivariant force fields

    Yi-Lun Liao, Tess Smidt, Muhammed Shuaibi, and Abhishek Das. Generalizing denoising to non-equilibrium structures improves equivariant force fields. arXiv preprint arXiv:2403.09549,

    arXiv

Show all 20 references
  1. [17]

    Geodiff: A geometric diffusion model for molecular conformation generation

    Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923,

    arXiv

  2. [18]

    Cong Fu, Keqiang Yan, Limei Wang, Wing Yee Au, Michael Curtis McThrow, Tao Komikado, Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and Shuiwang Ji

    URL https://arxiv.org/abs/2203.17003. Cong Fu, Keqiang Yan, Limei Wang, Wing Yee Au, Michael Curtis McThrow, Tao Komikado, Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and Shuiwang Ji. A latent diffusion model for protein structure generation. In Learning on Graphs Conference,...

    arXiv

  3. [19]

    org/abs/2001.09382

    URL https://arxiv. org/abs/2001.09382. Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation,

    2001 arXiv

  4. [20]

    16 A Appendix A.1 Metrics for Geometry Optimization Average Energy Minimization Percentage

    URL https://arxiv.org/abs/2102.01189. 16 A Appendix A.1 Metrics for Geometry Optimization Average Energy Minimization Percentage. This metric evaluates the extent to which the energy is reduced by the MLIP-optimized structure when compared to the DFT-optimized geometry: pctT =...

    arXiv

  5. [2000]

    Gradual optimization learning for conformational energy minimization

    Artem Tsypin, Leonid Ugadiarov, Kuzma Khrabrov, Alexander Telepov, Egor Rumiantsev, Alexey Skrynnik, Aleksandr I Panov, Dmitry Vetrov, Elena Tutubalina, and Artur Kadurin. Gradual optimization learning for conformational energy minimization. arXiv preprint arXiv:2311.06295,

    arXiv

  6. [2016]

    Fast graph representation learning with pytorch geometric

    Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428,

    1903 arXiv

  7. [2019]

    Fast quantum property prediction via deeper 2d and 3d graph networks

    Meng Liu, Cong Fu, Xuan Zhang, Limei Wang, Yaochen Xie, Hao Yuan, Youzhi Luo, Zhao Xu, Shenglong Xu, and Shuiwang Ji. Fast quantum property prediction via deeper 2d and 3d graph networks. arXiv preprint arXiv:2106.08551, 2021a. Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan La...

    arXiv

  8. [2020]

    Pre-training via denoising for molecular property prediction

    Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133,

    arXiv

  9. [2021]

    Ogb-lsc: A large-scale challenge for machine learning on graphs

    Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430,

    arXiv

  10. [2022]

    Molecule3d: A benchmark for predicting 3d geometries from molecular graphs

    Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs. arXiv preprint arXiv:2110.01717,

    arXiv

  11. [2023]

    Open catalyst 2020 (oc20) dataset and community challenges

    Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, et al. Open catalyst 2020 (oc20) dataset and community challenges. Acs Catalysis, 11(10):6059–6072,

    2020

  12. [2024]

    Artificial intelligence for science in quantum, atomistic, and continuum systems

    Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, ..., and Shuiwang Ji. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.