REVIEW 4 major objections 4 minor 2 cited by
GEOM-Drugs Revisited: Toward More Chemically Accurate Benchmarks for 3D Molecule Generation
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper claims that standard stability scores for 3D drug-like molecule generation are inflated by a valency-counting bug and an inconsistent force-field benchmark, and that correcting them lowers reported scores while preserving model…
desk verdict A credible, useful bug report on valency evaluation in GEOM-Drugs, wrapped around a comparative flow-vs-diffusion claim that the reported experiments do not support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the aromatic-dependent valency tuple: instead of a single valency per element and charge, each atom is described by $($element, formal charge, $n_{\mathrm{arom}}$, $v_{\mathrm{other}})$, where $n_{\mathrm{arom}}$ is the number of aromatic bonds and $v_{\mathrm{other}}$ is the total bond order from non-aromatic bonds, with the allowed tuples learned from the cleaned GEOM-Drugs set. This tuple is what distinguishes chemically valid aromatic environments, such as benzene carbon in the configuration $(2,1)$ or $(3,0)$, from invalid ones that the old table accepted. The second mechanism is the GFN2-xTB relaxation benchmark: each generated structure is re-optimized with GFN2-xTB and scored by the relaxation energy $\Delta E_{\mathrm{relax}}$ and by average changes in bond lengths, bond angles, and torsion angles between the raw and relaxed structures, giving an interpretable geometric and energetic measure of how far a model's output sits from the reference energy landscape.
What would settle it
Relax the same 5,000 generated molecules per model with a higher-level method such as DFT or coupled-cluster and compare model rankings by average relaxation energy; if the ordering changes, or if a molecule accepted by the aromatic-dependent valency table is found experimentally or computationally unstable, the corrected benchmark's conclusions would be overturned. A simpler check is whether the valency table reproduces all stable valency patterns in an independent large drug-like dataset.
Extended reading notes
Core claim
The paper's central claim is that the 'molecular stability' scores reported for GEOM-Drugs generative models are artificially high and chemically unreliable. The evidence is diagnostic: fixing the aromatic-bond rounding bug alone drops stability from roughly 0.94–0.99 to roughly 0.45–0.63, while replacing the lookup table with one that records each atom by element, formal charge, number of aromatic bonds, and non-aromatic bond order produces scores only 1–3 points below the published ones. The same framework then computes GFN2-xTB relaxation energies and internal-coordinate displacements, which show that diffusion-based models already outperform MMFF-optimized structures in alignment with the reference energy landscape. On that benchmark a consistent performance gap between flow-matching and diffusion-based models appears, even when the architecture family is the same. The paper frames the result as a call to abandon inherited buggy metrics and to adopt chemically grounded preprocessing and evaluation.
Load-bearing premise
The benchmark assumes GFN2-xTB is an accurate enough energy model for drug-like molecules that relaxing to a GFN2-xTB minimum is a meaningful measure of generated-molecule quality; if GFN2-xTB misjudges the relevant chemistry, the corrected scores and rankings could change.
Editorial extensions
If this is right
- Published stability numbers for models using the MiDi-style valency code should be read as upper bounds; the corrected aromatic-dependent lookup table lowers them by 1–3 percentage points.
- Retraining on a fully kekulized GEOM-Drugs version raises connected validity by about 5% for four of the six evaluated models, so the representation of aromatic bonds affects what the stability metric rewards.
- MMFF-based relaxation benchmarks are no longer suitable for ranking GEOM-Drugs models, since current models align closer to GFN2-xTB than MMFF does; MMFF remains useful only as a coarse structural-outlier filter.
- GFN2-xTB energy and geometry scoring reveals a flow-matching versus diffusion gap that did not surface under older metrics, so future benchmark comparisons should report relaxation energy and internal-coordinate displacements.
Reading between the lines
- Inference: the same aromatic-dependent valency treatment is likely needed on any aromatic-rich benchmark; a valency table learned only from GEOM-Drugs may not cover rare but viable chemistries in broader drug space.
- Inference: if GFN2-xTB relaxation energy is accepted as a quality signal, it could plausibly be used as a training-time objective or selection criterion for generated molecules, which the paper does not itself test.
- Inference: a testable next step is to run the same benchmark with a higher-level reference such as DFT on a subset of generated molecules; stable rankings across levels of theory would validate GFN2-xTB as a cheap benchmark standard.
- Inference: because the corrected stability metric drops scores by only 1–3 points for most models, small differences in future reported stability values should be treated cautiously unless the full uncertainty is reported.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper revisits the GEOM-Drugs benchmark for 3D molecule generation and argues that current evaluation protocols overstate model quality. The authors identify a bond-order counting bug in the widely used MiDi stability implementation (aromatic bonds rounded to 1 instead of 1.5), show that the associated valency lookup tables contain chemically implausible entries, and document that these errors propagated to several subsequent models. They propose an aromatic-dependent valency table, a kekulized and filtered version of GEOM-Drugs, and a GFN2-xTB-based geometry and energy benchmark, and they retrain or re-evaluate six models. The main quantitative claims are that corrected stability scores are 1-3% lower than originally reported, that retraining on kekulized data improves validity for most models, and that GFN2-xTB evaluations reveal a consistent flow-matching versus diffusion performance gap.
Significance. The paper's bug analysis and valency-table corrections are concrete and likely to be useful to the field; if adopted, the proposed pipeline would remove a known numerical artifact from a widely used metric. The authors also release a filtered dataset, processing scripts, and an implementation of the corrected stability metric, which strengthens reproducibility. The GFN2-xTB benchmark is a sensible response to the mismatch between MMFF-based evaluation and GFN2-xTB-optimized reference data, and the MMFF-to-GFN2-xTB baseline row in Table 2 neatly illustrates why that mismatch matters. However, the paper's headline comparative finding about flow-matching versus diffusion is not established by the reported experiments, because the relevant models differ in capacity and sampling budget.
major comments (4)
- [GFN2-xTB energy-based geometry benchmark, Table 2] The claim that corrected metrics reveal 'a consistent performance gap between flow-matching and diffusion-based models—even when the underlying architecture remains the same' is not supported by the reported experiments. The table caption states that diffusion models use 500 sampling steps while flow-matching models use 100 steps, and the text states that Megalodon-flow is a smaller 'quick' network and that SemlaFlow is a retrained variant. The 8-fold difference in mean Delta_E_relax between Megalodon (5.76 kcal/mol) and Megalodon-flow (46.9 kcal/mol), and the elevated SemlaFlow value (91.0 kcal/mol), could therefore be explained by model capacity or inference-time compute rather than by the generative paradigm. The authors should either benchmark the models at matched sampling budgets and comparable capacities, or explicitly withdraw the paradigm-level interpretation.
- [GFN2-xTB energy-based geometry benchmark, around Table 2] The choice of GFN2-xTB as the reference energy is justified only by the fact that GEOM-Drugs geometries were optimized with this method. Since generated molecules are scored by their relaxation energy and structural displacement to GFN2-xTB minima, the benchmark implicitly assumes that GFN2-xTB is an adequate description of the energy landscape for all generated chemistries. The paper does not check whether generated molecules have the same formal-charge or protonation-state distribution as the GEOM-Drugs reference, and it does not report any validation of GFN2-xTB against a higher-level method on a sample of generated molecules. Without such checks, the energy ranking in Table 2 is conditional on a reference model that may penalize valid but out-of-distribution chemistry.
- [Table 1 and footnotes] The claim that 'the relative rankings of the models remained largely consistent' is weakened by the fact that the table compares several modified variants rather than the original published models: JODO was retrained with the EQGAT-Diff objective using categorical diffusion, and Megalodon-flow is a retrained 'quick' variant that differs from the larger model in the original paper. The corrected numbers for these rows are therefore not direct re-evaluations of the models that originally reported the flawed metrics. The authors should either obtain or train the original configurations for these rows, or restrict the ranking-preservation claim to the models for which a direct comparison is available.
- [Solution and Table 5] The aromatic-dependent valency table is derived only from the filtered GEOM-Drugs set, and any tuple (element, number of aromatic bonds, formal charge, v_other) that is absent from the table is treated as invalid. The paper does not report what fraction of the 5,000 generated molecules per model contain atoms whose bonding environment is absent from Table 5, nor does it analyze whether those atoms are chemically reasonable. This matters because the reported 1-3% drop in stability under the corrected table could conflate genuine chemical invalidity with incomplete coverage of the lookup table. Reporting per-tuple coverage would make the metric's behavior transparent.
minor comments (4)
- [Section 'Solution', p. 7] 'their is no explicit modeling' should be 'there is no explicit modeling'; similarly, p. 11 contains 'semi-emperical' which should be 'semiempirical'.
- [Section 'Solution', cross-reference] The text refers to 'SI Table 5', but the table appears in Appendix I of the main supplement rather than in a separately numbered SI file; the cross-reference should be updated.
- [Table 5] The formatting of Table 5 is hard to read: entries such as '2, 1' for carbon with two aromatic bonds and charge 0 are not explained, so the reader cannot tell whether commas separate alternative allowed values or components of a tuple. A sentence defining the cell format would remove this ambiguity.
- [Repository link] The GitHub link should be versioned (for example, with a commit hash or release tag) so that the exact filtering scripts and valency tables can be cited reproducibly.
Circularity Check
No circularity found: the corrected valency tables and GFN2-xTB benchmarks are empirical, externally anchored evaluation criteria, not predictions derived from their own inputs.
full rationale
The paper's derivation chain is self-contained and empirically grounded rather than circular. (1) The valency-lookup corrections are explicitly empirical: valid valencies are 'defined as valencies observed in the training data,' and the refined (element, number of aromatic bonds, formal charge, valency) table is constructed from the filtered GEOM-Drugs set. Using a training-set-derived lookup table as an evaluation criterion is standard benchmarking practice; it is not a fitted parameter that is then renamed as a prediction. (2) The GFN2-xTB energy benchmark is anchored to an external reference—the GEOM-Drugs dataset was itself optimized with GFN2-xTB—so the benchmark choice is conditional on the dataset's documented generation protocol, not on the authors' models or fitted values. The reported relaxation energies and geometric displacements are computed from actual GFN2-xTB optimizations, not from the benchmark definition itself. (3) The comparisons against previously published models are re-evaluations on generated molecules; the stability scores in Table 1 change because the bugs and table entries differ, but this is a measurement correction, not a result forced by construction. (4) Self-citations to Megalodon (ref. 13), FlowMol (refs. 11–12), and SemlaFlow (ref. 9) are used as models under test, not as load-bearing evidence for the corrected framework; the correction arguments rest on chemical reasoning, RDKit-based kekulization, and the GFN2-xTB provenance of the dataset, not on these citations. The only substantive caveat—Table 2's unequal sampling budgets (500 diffusion steps vs. 100 flow-matching steps) and the smaller 'quick' Megalodon-flow architecture—is a confound for the flow-versus-diffusion gap claim, not a circularity, because it does not make any equation or metric equivalent to its own input. Accordingly, no circular step meets the evidentiary bar of exhibiting a specific reduction, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (1)
- Aromatic-dependent valency table (Table 5) =
Observed (n_arom, v_other) tuples per element and formal charge in the filtered GEOM-Drugs set
assumptions (4)
- domain assumption GFN2-xTB is an appropriate reference for evaluating drug-like molecule geometries and energies, and the GEOM-Drugs conformers are correctly optimized minima of this functional.
- domain assumption Formal charges are known or consistently assigned for generated molecules when running GFN2-xTB relaxation.
- domain assumption All remaining GEOM-Drugs molecules (after removing 0.18% fragmented ones) are chemically valid and their valencies are the correct ground truth.
- domain assumption Kekulization of all molecules in the prepared dataset is chemically unambiguous and does not alter molecular identity.
Cite this review
Pith. "Pith review of GEOM-Drugs Revisited: Toward More Chemically Accurate Benchmarks for 3D Molecule Generation." pith.science (2026). https://pith.science/paper/3HQEMKON
@misc{pith2026250500169,
author = {Pith},
title = {Pith review of: GEOM-Drugs Revisited: Toward More Chemically Accurate Benchmarks for 3D Molecule Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/3HQEMKON}},
note = {Machine review of arXiv:2505.00169}
}
read the original abstract
Deep generative models have shown significant promise in generating valid 3D molecular structures, with the GEOM-Drugs dataset serving as a key benchmark. However, current evaluation protocols suffer from critical flaws, including incorrect valency definitions, bugs in bond order calculations, and reliance on force fields inconsistent with the reference data. In this work, we revisit GEOM-Drugs and propose a corrected evaluation framework: we identify and fix issues in data preprocessing, construct chemically accurate valency tables, and introduce a GFN2-xTB-based geometry and energy benchmark. We retrain and re-evaluate several leading models under this framework, providing updated performance metrics and practical recommendations for future benchmarking. Our results underscore the need for chemically rigorous evaluation practices in 3D molecular generation. Our recommended evaluation methods and GEOM-Drugs processing scripts are available at https://github.com/isayevlab/geom-drugs-3dgen-evaluation.
Figures
Forward citations
Cited by 2 Pith papers
-
VEDA: 3D Molecular Generation via Variance-Exploding Diffusion with Annealing
VEDA generates 3D molecules with VE diffusion plus LMMSE preconditioning and an arcsin scheduler, reaching near-relaxed geometries with 100 sampling steps.
-
FlowMol3: Flow Matching for 3D De Novo Small-Molecule Generation
Combining self-conditioning, fake atoms, and late-stage geometry distortion lets a compact flow-matching model generate nearly always valid 3D drug-like molecules and match training-data chemistry better than existing...
Reference graph
Works this paper leans on
-
[1]
A 3D generative model for structure-based drug design
Luo, S.; Guan, J.; Ma, J.; Peng, J. A 3D generative model for structure-based drug design. Advances in Neural Information Processing Systems 2021, 34, 6229--6239
work page 2021
-
[2]
Bilodeau, C.; Jin, W.; Jaakkola, T.; Barzilay, R.; Jensen, K. F. Generative models for molecular discovery: Recent advances and challenges. Wiley Interdisciplinary Reviews: Computational Molecular Science 2022, 12, e1608
work page 2022
-
[3]
N.; Fourches, D.; Varnek, A.; Baskin, I
Cherkasov, A.; Muratov, E. N.; Fourches, D.; Varnek, A.; Baskin, I. I.; Cronin, M.; Dearden, J.; Gramatica, P.; Martin, Y. C.; Todeschini, R.; others QSAR modeling: where have you been? Where are you going to? Journal of medicinal chemistry 2014, 57, 4977--5010
work page 2014
-
[4]
Best practices for QSAR model development, validation, and exploitation
Tropsha, A. Best practices for QSAR model development, validation, and exploitation. Molecular informatics 2010, 29, 476--488
work page 2010
-
[5]
GFN2-xTB—An accurate and broadly parametrized self-consistent tight-binding quantum chemical method with multipole electrostatics and density-dependent dispersion contributions
Bannwarth, C.; Ehlert, S.; Grimme, S. GFN2-xTB—An accurate and broadly parametrized self-consistent tight-binding quantum chemical method with multipole electrostatics and density-dependent dispersion contributions. Journal of chemical theory and computation 2019, 15, 1652--1671
2019
-
[6]
T.; Coudert, F.-X.; Han, S.; Isayev, O.; Jain, A.; Walsh, A
Artrith, N.; Butler, K. T.; Coudert, F.-X.; Han, S.; Isayev, O.; Jain, A.; Walsh, A. Best practices in machine learning for chemistry. Nature chemistry 2021, 13, 505--508
work page 2021
-
[7]
Are the chemical structures in your QSAR correct? QSAR & combinatorial science 2008, 27, 1337--1345
Young, D.; Martin, T.; Venkatapathy, R.; Harten, P. Are the chemical structures in your QSAR correct? QSAR & combinatorial science 2008, 27, 1337--1345
work page 2008
-
[8]
GEOM, energy-annotated molecular conformations for property prediction and molecular generation
Axelrod, S.; Gomez-Bombarelli, R. GEOM, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data 2022, 9, 185
work page 2022
Show all 26 references
-
[9]
P.; Olsson, S
Irwin, R.; Tibo, A.; Janet, J. P.; Olsson, S. SemlaFlow--Efficient 3D Molecular Generation with Latent Attention and Equivariant Flow Matching. The 28th International Conference on Artificial Intelligence and Statistics
-
[10]
Navigating the design space of equivariant diffusion-based generative models for de novo 3d molecule generation
Le, T.; Cremer, J.; Noe, F.; Clevert, D.-A.; Sch \"u tt, K. Navigating the design space of equivariant diffusion-based generative models for de novo 3d molecule generation. arXiv preprint arXiv:2309.17296 2023,
2023 arXiv
-
[11]
Dunn, I.; Koes, D. R. Mixed continuous and categorical flow matching for 3d de novo molecule generation. ArXiv 2024, arXiv--2404
2024
-
[12]
Dunn, I.; Koes, D. R. Exploring Discrete Flow Matching for 3D De Novo Molecule Generation. ArXiv 2024,
2024
-
[13]
Reidenbach, D.; Nikitin, F.; Isayev, O.; Paliwal, S. G. Applications of Modular Co-Design for De Novo 3D Molecule Generation. NeurIPS 2024 Workshop on AI for New Drug Modalities
2024
-
[14]
Midi: Mixed graph and 3d denoising diffusion for molecule generation
Vignac, C.; Osman, N.; Toni, L.; Frossard, P. Midi: Mixed graph and 3d denoising diffusion for molecule generation. Joint European Conference on Machine Learning and Knowledge Discovery in Databases. 2023; pp 560--576
2023
-
[15]
Learning joint 2-d and 3-d graph diffusion models for complete molecule generation
Huang, H.; Sun, L.; Du, B.; Lv, W. Learning joint 2-d and 3-d graph diffusion models for complete molecule generation. IEEE Transactions on Neural Networks and Learning Systems 2024,
2024
-
[16]
E (n) equivariant normalizing flows
Garcia Satorras, V.; Hoogeboom, E.; Fuchs, F.; Posner, I.; Welling, M. E (n) equivariant normalizing flows. Advances in Neural Information Processing Systems 2021, 34, 4181--4192
2021
-
[17]
G.; Vignac, C.; Welling, M
Hoogeboom, E.; Satorras, V. G.; Vignac, C.; Welling, M. Equivariant diffusion for molecule generation in 3d. International conference on machine learning. 2022; pp 8867--8887
2022
-
[18]
Geometry-complete diffusion for 3D molecule generation and optimization
Morehead, A.; Cheng, J. Geometry-complete diffusion for 3D molecule generation and optimization. Communications Chemistry 2024, 7, 150
2024
-
[19]
Equivariant flow matching with hybrid probability transport for 3d molecule generation
Song, Y.; Gong, J.; Xu, M.; Cao, Z.; Lan, Y.; Ermon, S.; Zhou, H.; Ma, W.-Y. Equivariant flow matching with hybrid probability transport for 3d molecule generation. Advances in Neural Information Processing Systems 2023, 36, 549--568
2023
-
[20]
Geometric-facilitated denoising diffusion model for 3D molecule generation
Xu, C.; Wang, H.; Wang, W.; Zheng, P.; Chen, H. Geometric-facilitated denoising diffusion model for 3D molecule generation. Proceedings of the AAAI Conference on Artificial Intelligence. 2024; pp 338--346
2024
-
[21]
Equivariant neural diffusion for molecule generation
Cornet, F.; Bartosh, G.; Schmidt, M.; Andersson Naesseth, C. Equivariant neural diffusion for molecule generation. Advances in Neural Information Processing Systems 2024, 37, 49429--49460
2024
-
[22]
NEXT-MOL: 3d diffusion meets 1d language modeling for 3d molecule generation
Liu, Z.; Luo, Y.; Huang, H.; Zhang, E.; Li, S.; Fang, J.; Shi, Y.; Wang, X.; Kawaguchi, K.; Chua, T.-S. NEXT-MOL: 3d diffusion meets 1d language modeling for 3d molecule generation. arXiv preprint arXiv:2502.12638 2025,
2025 arXiv
-
[23]
Cremer, J.; Le, T.; No \'e , F.; Clevert, D.-A.; Sch \"u tt, K. T. PILOT: equivariant diffusion for pocket-conditioned de novo ligand generation with multi-objective guidance via importance sampling. Chemical Science 2024, 15, 14954--14967
2024
-
[24]
Geodiff: A geometric diffusion model for molecular conformation generation
Xu, M.; Yu, L.; Song, Y.; Shi, C.; Ermon, S.; Tang, J. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923 2022,
2022 arXiv
-
[25]
Energy windows for computed compound conformers: covering artefacts or truly large reorganization energies? Future Medicinal Chemistry 2019, 11, 97--118
Foloppe, N.; Chen, I.-J. Energy windows for computed compound conformers: covering artefacts or truly large reorganization energies? Future Medicinal Chemistry 2019, 11, 97--118
2019
-
[26]
M.; Deane, C
Buttenschoen, M.; Morris, G. M.; Deane, C. M. PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences. Chemical Science 2024, 15, 3130--3139 mcitethebibliography si.tex0000664000000000000000000001663115011527745010731 0ust...
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.