{"id":"a268b066-aa9d-4272-8a19-7600d0f05489","arxiv_id":"2505.00169","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A corrected evaluation framework for GEOM-Drugs, including a valency-counting fix and GFN2-xTB-based energy and geometry metrics, changes reported stability scores by up to several percent and reveals a flow-matching versus diffusion performance gap.","lead":"This paper finds and fixes errors in how AI-generated 3D molecules are graded, including a bond-counting bug that marked some bad molecules as stable. It also proposes a more reliable test based on the same quantum-chemistry method used to create the reference data, changing reported scores while mostly preserving model rankings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The flow-matching versus diffusion gap in Table 2 is confounded by model size and sampling budget, so the claim that corrected metrics reveal a paradigm gap is not established.","rationale":"The reader identified GFN2-xTB accuracy as the weakest assumption. I agree that this is a limitation, but the paper explicitly uses GFN2-xTB because it is the same level of theory used to generate GEOM-Drugs, so the energy benchmark is at least internally consistent and does not need to be a universal chemical ground truth for the core valency and MMFF arguments to hold. The more actionable, internally visible problem is the flow-matching versus diffusion comparison in Table 2. The paper itself discloses the unequal sampling steps and the smaller Megalodon-flow network, so the comparison is not controlled. This does not overturn the corrected evaluation framework or the identified valency bugs, but it does mean the paper's prominent paradigm-gap conclusion is not established. The reader's CONDITIONAL verdict remains appropriate, with the additional condition that the flow/diffusion claim be removed, qualified, or supported by matched-capacity and matched-budget experiments.","tokens_in":9785,"tokens_out":10887,"duration_ms":120863,"concrete_test":"Evaluate Megalodon-flow using the exact Megalodon architecture and parameter count, and run both Megalodon and Megalodon-flow at identical sampling budgets (e.g., 100, 200, and 500 steps), recomputing Table 2's mean Delta_E_relax on the same 5,000 valid molecules. If the gap persists at matched capacity and compute, the paradigm claim is supported; if it shrinks or reverses, the paper should retract or explicitly qualify the flow-matching versus diffusion gap claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most striking new result is that the GFN2-xTB energy benchmark reveals a consistent performance gap between flow-matching and diffusion models, even when the architecture is the same. This claim is not supported by the reported experiments. Table 2 explicitly states that diffusion models use 500 steps while flow-matching models use 100 steps, and the text admits that Megalodon-flow is a smaller 'quick' network necessitated by limited compute. SemlaFlow is also a retrained variant. Consequently, the 8-fold difference in mean Delta_E_relax between Megalodon (5.76 kcal/mol) and Megalodon-flow (46.9 kcal/mol) could be due to model capacity or inference-time compute rather than the evaluation framework. The same confound affects the elevated SemlaFlow value (91.0 kcal/mol). The valency-table corrections and the MMFF-versus-GFN2-xTB inconsistency are independent and credible, but the comparative conclusion that the corrected benchmark reveals a flow/diffusion gap does not follow from the data as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper revisits the GEOM-Drugs benchmark for 3D molecule generation and argues that current evaluation protocols overstate model quality. The authors identify a bond-order counting bug in the widely used MiDi stability implementation (aromatic bonds rounded to 1 instead of 1.5), show that the associated valency lookup tables contain chemically implausible entries, and document that these errors propagated to several subsequent models. They propose an aromatic-dependent valency table, a kekulized and filtered version of GEOM-Drugs, and a GFN2-xTB-based geometry and energy benchmark, and they retrain or re-evaluate six models. The main quantitative claims are that corrected stability scores are 1-3% lower than originally reported, that retraining on kekulized data improves validity for most models, and that GFN2-xTB evaluations reveal a consistent flow-matching versus diffusion performance gap.","tokens_in":9939,"tokens_out":5207,"duration_ms":49090,"significance":"The paper's bug analysis and valency-table corrections are concrete and likely to be useful to the field; if adopted, the proposed pipeline would remove a known numerical artifact from a widely used metric. The authors also release a filtered dataset, processing scripts, and an implementation of the corrected stability metric, which strengthens reproducibility. The GFN2-xTB benchmark is a sensible response to the mismatch between MMFF-based evaluation and GFN2-xTB-optimized reference data, and the MMFF-to-GFN2-xTB baseline row in Table 2 neatly illustrates why that mismatch matters. However, the paper's headline comparative finding about flow-matching versus diffusion is not established by the reported experiments, because the relevant models differ in capacity and sampling budget.","major_comments":[{"comment":"The claim that corrected metrics reveal 'a consistent performance gap between flow-matching and diffusion-based models—even when the underlying architecture remains the same' is not supported by the reported experiments. The table caption states that diffusion models use 500 sampling steps while flow-matching models use 100 steps, and the text states that Megalodon-flow is a smaller 'quick' network and that SemlaFlow is a retrained variant. The 8-fold difference in mean Delta_E_relax between Megalodon (5.76 kcal/mol) and Megalodon-flow (46.9 kcal/mol), and the elevated SemlaFlow value (91.0 kcal/mol), could therefore be explained by model capacity or inference-time compute rather than by the generative paradigm. The authors should either benchmark the models at matched sampling budgets and comparable capacities, or explicitly withdraw the paradigm-level interpretation.","section":"GFN2-xTB energy-based geometry benchmark, Table 2"},{"comment":"The choice of GFN2-xTB as the reference energy is justified only by the fact that GEOM-Drugs geometries were optimized with this method. Since generated molecules are scored by their relaxation energy and structural displacement to GFN2-xTB minima, the benchmark implicitly assumes that GFN2-xTB is an adequate description of the energy landscape for all generated chemistries. The paper does not check whether generated molecules have the same formal-charge or protonation-state distribution as the GEOM-Drugs reference, and it does not report any validation of GFN2-xTB against a higher-level method on a sample of generated molecules. Without such checks, the energy ranking in Table 2 is conditional on a reference model that may penalize valid but out-of-distribution chemistry.","section":"GFN2-xTB energy-based geometry benchmark, around Table 2"},{"comment":"The claim that 'the relative rankings of the models remained largely consistent' is weakened by the fact that the table compares several modified variants rather than the original published models: JODO was retrained with the EQGAT-Diff objective using categorical diffusion, and Megalodon-flow is a retrained 'quick' variant that differs from the larger model in the original paper. The corrected numbers for these rows are therefore not direct re-evaluations of the models that originally reported the flawed metrics. The authors should either obtain or train the original configurations for these rows, or restrict the ranking-preservation claim to the models for which a direct comparison is available.","section":"Table 1 and footnotes"},{"comment":"The aromatic-dependent valency table is derived only from the filtered GEOM-Drugs set, and any tuple (element, number of aromatic bonds, formal charge, v_other) that is absent from the table is treated as invalid. The paper does not report what fraction of the 5,000 generated molecules per model contain atoms whose bonding environment is absent from Table 5, nor does it analyze whether those atoms are chemically reasonable. This matters because the reported 1-3% drop in stability under the corrected table could conflate genuine chemical invalidity with incomplete coverage of the lookup table. Reporting per-tuple coverage would make the metric's behavior transparent.","section":"Solution and Table 5"}],"minor_comments":[{"comment":"'their is no explicit modeling' should be 'there is no explicit modeling'; similarly, p. 11 contains 'semi-emperical' which should be 'semiempirical'.","section":"Section 'Solution', p. 7"},{"comment":"The text refers to 'SI Table 5', but the table appears in Appendix I of the main supplement rather than in a separately numbered SI file; the cross-reference should be updated.","section":"Section 'Solution', cross-reference"},{"comment":"The formatting of Table 5 is hard to read: entries such as '2, 1' for carbon with two aromatic bonds and charge 0 are not explained, so the reader cannot tell whether commas separate alternative allowed values or components of a tuple. A sentence defining the cell format would remove this ambiguity.","section":"Table 5"},{"comment":"The GitHub link should be versioned (for example, with a commit hash or release tag) so that the exact filtering scripts and valency tables can be cited reproducibly.","section":"Repository link"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution is the bug report. MiDi's valency code counted aromatic bonds as 1 instead of 1.5 and then used the resulting chemically permissive table to judge stability. The paper shows this exact implementation was copied into several later models, and it quantifies the effect. I believe that. The aromatic-dependent valency table (n_arom, v_other) is a sensible fix, and the decision to filter 0.18% of GEOM-Drugs where GFN2-xTB fragmented molecules is defensible. The GFN2-xTB geometry/energy benchmark is also well motivated: if you train on GFN2-xTB conformers, evaluating relaxation against MMFF is a mismatch. That part of the paper is solid and should be adopted.\n\nThe soft spot is the flow-vs-diffusion conclusion. Table 2 shows a large gap, but the table itself says diffusion models use 500 steps and flow-matching models use 100 steps, and Megalodon-flow is a smaller 'quick' network. So the gap between Megalodon and Megalodon-flow is not 'the same architecture,' and the difference could easily be sampling compute or model capacity. The paper states the gap appears 'even when the underlying architecture remains the same,' but the reported experiments do not support that sentence. This is the kind of claim that will get cited, so it needs to be fixed before publication, either by removing it or by running controlled comparisons at matched steps and capacity.\n\nLesser points: the retrained variants (JODO with a different training objective, Megalodon-flow quick) mean Table 1's right section is not a direct comparison to the original papers; the text should mark which rows are modified. The energy benchmark assumes GFN2-xTB as the reference landscape, which is reasonable given the dataset, though a small DFT sanity subset would strengthen it. The valency table derived from the same dataset is standard lookup-table practice and not circular in a damaging way.\n\nWho this is for: anyone benchmarking or building 3D molecule generators. It deserves a serious referee and probably acceptance with revisions. I'd cite it for the bug alone.","headline":"A credible, useful bug report on valency evaluation in GEOM-Drugs, wrapped around a comparative flow-vs-diffusion claim that the reported experiments do not support.","tokens_in":10531,"tokens_out":3499,"would_cite":true,"duration_ms":35182,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that standard stability scores for 3D drug-like molecule generation are inflated by a valency-counting bug and an inconsistent force-field benchmark, and that correcting them lowers reported scores while preserving model…","keywords":["3D molecule generation","GEOM-Drugs","molecular stability","valency","GFN2-xTB","energy benchmark","evaluation metrics","flow matching"],"falsifier":"Relax the same 5,000 generated molecules per model with a higher-level method such as DFT or coupled-cluster and compare model rankings by average relaxation energy; if the ordering changes, or if a molecule accepted by the aromatic-dependent valency table is found experimentally or computationally unstable, the corrected benchmark's conclusions would be overturned. A simpler check is whether the valency table reproduces all stable valency patterns in an independent large drug-like dataset.","tokens_in":9560,"feed_emoji":"🧪","tokens_out":7929,"duration_ms":75970,"temperature":0.7,"pith_summary":"This paper argues that widely used evaluation protocols for 3D generation of drug-like molecules on the GEOM-Drugs benchmark overstate how well models perform. It traces the inflation to three fixable flaws: aromatic bonds counted with a valency of 1 instead of 1.5 in a widely reused implementation, a valency lookup table that accepts chemically impossible configurations such as neutral trivalent carbon, and energy evaluations using the MMFF force field even though the reference dataset was optimized with GFN2-xTB. The paper supplies corrected valency tables, removes data fragments left by failed geometry optimizations, and proposes a geometry and relaxation-energy benchmark computed with GFN2-xTB. Re-evaluated models lose 1–3 percentage points of molecular stability under the corrected valency scheme, and retraining on a kekulized dataset raises validity by roughly 5% for most tested models. The rankings of the models survive the correction, but a previously obscured gap between flow-matching and diffusion-based models becomes visible.","feed_headline":"Valency bug inflated 3D molecule generation scores","feed_subtitle":"Aromatic-bond counting and force-field fixes change reported accuracy but preserve model rankings.","key_machinery":"The central object is the aromatic-dependent valency tuple: instead of a single valency per element and charge, each atom is described by $($element, formal charge, $n_{\\mathrm{arom}}$, $v_{\\mathrm{other}})$, where $n_{\\mathrm{arom}}$ is the number of aromatic bonds and $v_{\\mathrm{other}}$ is the total bond order from non-aromatic bonds, with the allowed tuples learned from the cleaned GEOM-Drugs set. This tuple is what distinguishes chemically valid aromatic environments, such as benzene carbon in the configuration $(2,1)$ or $(3,0)$, from invalid ones that the old table accepted. The second mechanism is the GFN2-xTB relaxation benchmark: each generated structure is re-optimized with GFN2-xTB and scored by the relaxation energy $\\Delta E_{\\mathrm{relax}}$ and by average changes in bond lengths, bond angles, and torsion angles between the raw and relaxed structures, giving an interpretable geometric and energetic measure of how far a model's output sits from the reference energy landscape.","core_discovery":"The paper's central claim is that the 'molecular stability' scores reported for GEOM-Drugs generative models are artificially high and chemically unreliable. The evidence is diagnostic: fixing the aromatic-bond rounding bug alone drops stability from roughly 0.94–0.99 to roughly 0.45–0.63, while replacing the lookup table with one that records each atom by element, formal charge, number of aromatic bonds, and non-aromatic bond order produces scores only 1–3 points below the published ones. The same framework then computes GFN2-xTB relaxation energies and internal-coordinate displacements, which show that diffusion-based models already outperform MMFF-optimized structures in alignment with the reference energy landscape. On that benchmark a consistent performance gap between flow-matching and diffusion-based models appears, even when the architecture family is the same. The paper frames the result as a call to abandon inherited buggy metrics and to adopt chemically grounded preprocessing and evaluation.","pith_inferences":["Inference: the same aromatic-dependent valency treatment is likely needed on any aromatic-rich benchmark; a valency table learned only from GEOM-Drugs may not cover rare but viable chemistries in broader drug space.","Inference: if GFN2-xTB relaxation energy is accepted as a quality signal, it could plausibly be used as a training-time objective or selection criterion for generated molecules, which the paper does not itself test.","Inference: a testable next step is to run the same benchmark with a higher-level reference such as DFT on a subset of generated molecules; stable rankings across levels of theory would validate GFN2-xTB as a cheap benchmark standard.","Inference: because the corrected stability metric drops scores by only 1–3 points for most models, small differences in future reported stability values should be treated cautiously unless the full uncertainty is reported."],"forward_implications":["Published stability numbers for models using the MiDi-style valency code should be read as upper bounds; the corrected aromatic-dependent lookup table lowers them by 1–3 percentage points.","Retraining on a fully kekulized GEOM-Drugs version raises connected validity by about 5% for four of the six evaluated models, so the representation of aromatic bonds affects what the stability metric rewards.","MMFF-based relaxation benchmarks are no longer suitable for ranking GEOM-Drugs models, since current models align closer to GFN2-xTB than MMFF does; MMFF remains useful only as a coarse structural-outlier filter.","GFN2-xTB energy and geometry scoring reveals a flow-matching versus diffusion gap that did not surface under older metrics, so future benchmark comparisons should report relaxation energy and internal-coordinate displacements."],"supporting_citations":[{"why":"Supplies the GEOM-Drugs dataset whose GFN2-xTB-optimized conformers define the reference energy landscape for all corrected benchmarks.","marker":"8"},{"why":"Defines the GFN2-xTB semi-empirical method used both to produce the reference dataset and to compute the new relaxation-energy and geometry metrics.","marker":"5"},{"why":"Introduced the atom-level valency stability metric that the paper argues is buggy in later reimplementations and re-derives here.","marker":"17"},{"why":"MiDi is the implementation whose aromatic-bond valency rounding to 1 and inflated lookup table propagated to later models.","marker":"14"},{"why":"SemlaFlow is one of the re-evaluated models whose stability and energy numbers must be corrected under the new protocol.","marker":"9"},{"why":"EQGAT-Diff is another re-evaluated model showing how the flawed stability implementation was reused in subsequent work.","marker":"10"},{"why":"FlowMol2 is among the re-evaluated models trained on the original dataset and used to quantify the effects of kekulization.","marker":"12"},{"why":"Megalodon variants are re-evaluated and contribute the flow-matching results that expose the flow-versus-diffusion gap.","marker":"13"},{"why":"Provides evidence that MMFF relaxation energies carry 15–20 kcal/mol errors relative to higher-level methods, undermining MMFF as a benchmark.","marker":"25"},{"why":"Supports retaining MMFF only as a coarse energy-based outlier filter rather than a ranking metric.","marker":"26"}],"fun_headline_variants":["Aromatic bond bug drops GEOM-Drugs stability from 0.99 to 0.45","Valency fix exposes inflated scores in 3D molecule generation","GFN2-xTB benchmark reveals true performance of 3D generators","Diffusion models outpace flow-matching on corrected GEOM-Drugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes GFN2-xTB is an accurate enough energy model for drug-like molecules that relaxing to a GFN2-xTB minimum is a meaningful measure of generated-molecule quality; if GFN2-xTB misjudges the relevant chemistry, the corrected scores and rankings could change.","fun_headline_variants_meta":{"raw":{"variants":["Aromatic bond bug drops GEOM-Drugs stability from 0.99 to 0.45","Valency fix exposes inflated scores in 3D molecule generation","GFN2-xTB benchmark reveals true performance of 3D generators","Diffusion models outpace flow-matching on corrected GEOM-Drugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000912,"raw_usage":{"total_tokens":3893,"prompt_tokens":894,"completion_tokens":2999,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2914}},"tokens_in":510,"tokens_out":2999,"duration_ms":22187,"temperature":1.0,"reasoning_tokens":2914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:50:03.908949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Relax the same 5,000 generated molecules per model with a higher-level method such as DFT or coupled-cluster and compare model rankings by average relaxation energy; if the ordering changes, or if a molecule accepted by the aromatic-dependent valency table is found experimentally or computationally unstable, the corrected benchmark's conclusions would be overturned. A simpler check is whether the valency table reproduces all stable valency patterns in an independent large drug-like dataset.","supporting_citations":[{"cited_title":"GEOM, energy-annotated molecular conformations for property prediction and molecular generation","cited_arxiv_id":null,"evidence_quote":"Supplies the GEOM-Drugs dataset whose GFN2-xTB-optimized conformers define the reference energy landscape for all corrected benchmarks."},{"cited_title":"G.; Vignac, C.; Welling, M","cited_arxiv_id":null,"evidence_quote":"Introduced the atom-level valency stability metric that the paper argues is buggy in later reimplementations and re-derives here."},{"cited_title":"Midi: Mixed graph and 3d denoising diffusion for molecule generation","cited_arxiv_id":null,"evidence_quote":"MiDi is the implementation whose aromatic-bond valency rounding to 1 and inflated lookup table propagated to later models."},{"cited_title":"P.; Olsson, S","cited_arxiv_id":null,"evidence_quote":"SemlaFlow is one of the re-evaluated models whose stability and energy numbers must be corrected under the new protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FlowMol2 is among the re-evaluated models trained on the original dataset and used to quantify the effects of kekulization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Megalodon variants are re-evaluated and contribute the flow-matching results that expose the flow-versus-diffusion gap."},{"cited_title":"Energy windows for computed compound conformers: covering artefacts or truly large reorganization energies? Future Medicinal Chemistry 2019, 11, 97--118","cited_arxiv_id":null,"evidence_quote":"Provides evidence that MMFF relaxation energies carry 15–20 kcal/mol errors relative to higher-level methods, undermining MMFF as a benchmark."},{"cited_title":"M.; Deane, C","cited_arxiv_id":null,"evidence_quote":"Supports retaining MMFF only as a coarse energy-based outlier filter rather than a ranking metric."}],"review_version":1}