{"id":"8136fffb-3244-4f19-931f-99bb0b4c298e","arxiv_id":"2501.14039","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AtomProNet is a new open-source tool that bundles data collection, DFT job submission, and dataset preparation for machine learning interatomic potentials, demonstrated on alumina.","lead":"This paper introduces AtomProNet, an open-source Python package that automates pulling crystal structures from databases, generating DFT training data, and preparing it for machine learning interatomic potential training. A case study on alumina compares two machine learning potentials against classical reactive potentials on accuracy and computational speed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MLIP-versus-classical benchmark is confounded by in-distribution fitting: the MLIPs are trained on thousands of DFT snapshots of the very alumina defect/strained configurations later used for evaluation, while ReaxFF and COMB3 are generic potentials not fitted to those data, so the accuracy…","rationale":"The reader's weakest_assumption identifies exactly the load-bearing issue in the paper's central claim: the MLIP-versus-classical benchmark is not a fair test because the MLIPs are trained on in-distribution data while the classical potentials are generic and not fitted to the same reference set. My independent reading of the manuscript reaches the same conclusion, and I also flag the secondary dataset-size inconsistency as a supporting symptom of insufficient care in reporting the data pipeline. I did not find a more fundamental flaw: the software contribution—an open-source package that automates structure retrieval, DFT job submission, batch data processing, and MLIP training—is plausible, the code is made available, and the pipeline modules are described in enough detail to be reproducible. The survey claim in the abstract is unsupported by any reported data, but that is not the primary basis for the central claim about MLIP accuracy. The appropriate verdict remains CONDITIONAL: the paper should be accepted only after the authors reposition the benchmark as a software demonstration with an out-of-distribution test, or provide the missing survey data and a more balanced comparison. Since the reader's verdict already reflects this condition, no change to the verdict is needed.","tokens_in":17130,"tokens_out":3055,"duration_ms":28706,"concrete_test":"Reproduce the benchmark with a genuinely out-of-distribution test set: generate DFT reference data for configurations not present in the training manifold—for example, 300 K AIMD trajectories of the perfect and vacancy supercells, amorphous or gamma-Al2O3, alumina surfaces, or random atomic displacements with non-volumetric strain modes—hold these structures out of training, and recompute energy/force MAEs and the Table 1 properties for Allegro, MACE, ReaxFF, and COMB3. Alternatively, retune ReaxFF and COMB3 parameters against the same 14,000-structure training set (or use their best-published alumina parameterizations) and re-run the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is that Allegro and MACE are more accurate and practical than ReaxFF and COMB3 for alumina. That claim rests on a comparison that is not apples-to-apples. Section 3.1 describes generating 12,000 hydrostatically strained alumina supercell cases, while Section 3.2 reports training on data from 16,000 DFT structures split into 14,000 training, 1,500 test, and 500 validation samples. Figure 3 reports parity plots and error distributions on the 500-structure validation set, which is drawn from the same configuration distribution as the training set. Table 1 then evaluates lattice constants, elastic constants, cohesive energies, and vacancy formation energies for the same alumina phases and defect types used in training. Thus the MLIPs have been fit to a large, dense set of in-distribution labels for exactly the properties and configurations being benchmarked. ReaxFF and COMB3, by contrast, are generic reactive potentials whose parameters were fixed by previous, separate fitting efforts and were not re-optimized against the DFT reference data used here. The accuracy gap in Table 1 is therefore the expected outcome of data-specific fitting, not evidence of a general superiority of MLIPs over classical potentials. An additional internal inconsistency—12,000 generated cases in Section 3.1 versus 16,000 structures in Section 3.2—further undermines confidence in the dataset description. The computational performance comparison (Fig. 4) is also sensitive to implementation details, such as QeQ being applied every time step versus every 1,000 steps in the supplement, and to whether MLIP inference is measured on CPU or GPU, so the 'practicality' claim is less secure than the text suggests. The central software-pipeline contribution can be valid even if this benchmark is not, but the paper's headline accuracy claim is not supported by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AtomProNet, an open-source Python package that automates the data pipeline for machine-learning interatomic potentials (MLIPs), including retrieving structures from materials databases, generating and submitting DFT jobs, batch-processing outputs, and preparing training/validation data for neural network potentials. The authors motivate the package with a global survey (not detailed in the manuscript) and demonstrate it on an alumina case study, in which they train Allegro and MACE models and compare their accuracy and computational speed against the classical reactive potentials ReaxFF and COMB3. The stated goal is to lower access barriers for non-specialist researchers by streamlining the workflow from ab initio data to MLIP training and evaluation.","tokens_in":17477,"tokens_out":5154,"duration_ms":45185,"significance":"If the software works as described, it is a useful and welcome contribution to a real practical bottleneck: the transition from DFT datasets to MLIP training is still labor-intensive and fragmented. The provision of a public GitHub repository and a Colab demo is a genuine strength that supports reproducibility and lowers adoption costs. The paper also gives a clear, well-illustrated overview of the components of MLIP workflows, which is helpful didactic material. However, the quantitative case study—the comparison of MLIPs with classical potentials—is not a controlled benchmark: the MLIPs are fitted to DFT data for the same material and structure families used in the comparison, while ReaxFF and COMB3 are generic potentials not fitted to this dataset. The accuracy conclusions drawn from this comparison are therefore weaker than the text suggests. The survey, which is presented as a design driver, is not documented at all. These issues do not invalidate the software contribution but they do require substantial revision of the claims and the presentation of the case study.","major_comments":[{"comment":"There is an unresolved inconsistency in the dataset description. Section 3.1 states that hydrostatic straining of the relaxed 80-atom alumina supercell with a step size of 0.01% over ±10% generates 12,000 cases, while Section 3.2 states that energy, force, stress, and lattice data from 16,000 DFT structures were used, split into 14,000 training, 1,500 test, and 500 validation samples. The relationship between the 12,000 generated structures and the 16,000 DFT structures is not explained. This inconsistency must be resolved and the data-generation procedure described precisely (including how 12,000 arises from the stated strain range and step, and how the validation set relates to the generated configurations). Without this, the reported parity plots and training/test splits are not reproducible.","section":"§3.1 and §3.2"},{"comment":"The accuracy comparison between MLIPs and classical potentials is confounded by in-distribution fitting. The Allegro and MACE models are trained on DFT energies and forces for alumina structures that include the same vacancy and strained configurations later used for validation (Section 3.2), and the Table 1 property evaluation concerns the same alumina phases and defect types. ReaxFF and COMB3, by contrast, are generic potentials whose parameters were fixed by previous fitting efforts and were not re-optimized against the DFT reference data used here. The accuracy gap in Table 1 is therefore the expected result of data-specific fitting, not evidence of a general superiority of MLIPs over classical potentials. To support the comparative claim, the authors should either benchmark on out-of-distribution structures (e.g., different phases, surfaces, or thermodynamic conditions not in the training set) or fit a classical potential to the same DFT data. At minimum, the text should explicitly acknowledge this limitation and soften the corresponding conclusions.","section":"§4.1, Table 1"},{"comment":"The definition of the reported 'vacancy formation energies' appears to be incorrect or at least unclear. The tabulated values for O vacancy, Al vacancy, and Al-O vacancy are negative and very close to the cohesive energy per atom (e.g., -7.343, -7.296, -7.312 eV/atom for DFT). A defect formation energy is normally a positive quantity defined as the energy difference between the defective and perfect cells with appropriate chemical-potential corrections. As presented, these numbers look like total energies per atom of the defective cells, not formation energies. In addition, the DFT reference values in Table 1 are cited to Refs. [100–102], which concern high-pressure alumina phases and MgAl2O4, not the ground-state corundum elastic constants and defect energies tabulated here. The authors need to provide a precise definition of each property, the exact formula used, and references that actually report those values.","section":"§4.1, Table 1"},{"comment":"The paper states that a global survey was conducted and that the survey responses were implemented to design AtomProNet, yet no information is given about the survey instrument, sampling, number of respondents, geographic/demographic distribution, or any quantitative or qualitative results. The claimed link between the survey and the software design is therefore unverifiable. The authors should either report the survey methodology and a summary of the findings (in the main text or supplementary material) or remove the claim that the survey guided the design. This is a load-bearing point because the survey is presented as the motivation for the software in both the abstract and the introduction.","section":"§1 and Abstract"}],"minor_comments":[{"comment":"The caption is garbled: the subplot letters are repeated and inconsistent (e.g., '(b)' appears twice, and the cumulative distribution plots are labeled (c) and (d) in the text but (e)–(h) in the caption). Please renumber the panels to match the in-text references.","section":"Figure 3 caption"},{"comment":"The sentence 'The electronic energies cut-off, the kinetic-energy cutoff, Monkhorst–Pack k-points and width of Gaussian smearing were 10–6 eV, 1 meV/atom, 4 × 4 × 4, and 0.026eV, respectively' is confusing: 'electronic energies cut-off' likely means the electronic energy convergence criterion, and 'kinetic-energy cutoff' is the plane-wave cutoff. Please clarify the terms and correct the units.","section":"§3.1"},{"comment":"The phrase 'simulation time per computational time' is imprecise. Please define the performance metric explicitly (e.g., timesteps per second or nanoseconds of simulated time per wall-clock hour) and state the hardware and software version details for the timing runs.","section":"§4.1"},{"comment":"The abstract contains the typo 'start-of-the-art' (should be 'state-of-the-art'). Minor wording issues of this kind appear throughout the manuscript; a careful proofread is recommended.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The core deliverable—the AtomProNet software—appears to be a useful and relevant contribution, and the open release of the code is commendable. My main concern is that the paper's quantitative claims go beyond what the benchmark actually demonstrates. The in-distribution comparison with generic classical potentials, the unresolved 12,000/16,000 dataset discrepancy, and the questionable definition of vacancy formation energies should be addressed before the paper can be accepted. I would also encourage the editor to consider whether the survey results, which are invoked prominently but never shown, can be added in a supplementary document; if not, the text should be revised to remove the claim that the survey informed the design. With these revisions, the software-focused contribution could be a solid fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The software is a real contribution; the benchmark is not. AtomProNet automates a genuinely tedious part of the MLIP workflow—pulling structures from Materials Project, writing and submitting VASP jobs, collecting energies/forces/stresses, and formatting them for Allegro and MACE training. The code and a Colab demo are up on GitHub, and the modular design looks sensible. That alone is worth something for groups that want to train their own potentials.\n\nWhat the paper does not deliver is a fair test of MLIPs versus classical potentials. The MLIPs are trained on 14,000 DFT structures that include hydrostatically strained alumina and vacancy configurations, and then evaluated on a 500-structure validation set from the same distribution, plus property tables for the same phases and defect types. ReaxFF and COMB3 are generic potentials not fitted to this DFT reference. So the accuracy gap is the expected outcome of data-specific fitting, not evidence that MLIPs are generally superior. The computational performance comparison has a similar issue: Allegro is faster in the main text, but when QeQ is applied every 1000 steps instead of every step, ReaxFF and COMB3 outperform Allegro. That is a legitimate implementation detail, but it means the 'practicality' claim is sensitive to choices that the text does not foreground.\n\nThere are also two internal inconsistencies that should have been caught. Section 3.1 says 12,000 hydrostatically strained cases were generated; Section 3.2 says 16,000 DFT structures were used. The survey is described as the design basis for the software, but no survey questions, response counts, or analysis appear in the paper or supplement. I cannot verify that the design actually reflects survey input.\n\nNone of this kills the software contribution. Reposition the paper as a pipeline demonstration, report the survey data or cut the claim, fix the dataset numbers, and add an out-of-distribution test—for example, evaluating on a different phase or on dynamics not in the training set—and the comparison to classical potentials becomes either fair or at least explicitly qualified. As it stands, the central accuracy claim is not supported, but the package itself deserves referee time.","headline":"AtomProNet is a genuinely useful automation tool, but the paper's accuracy and practicality claims are undermined by an in-distribution benchmark and an apples-to-oranges comparison.","tokens_in":18057,"tokens_out":2757,"would_cite":false,"duration_ms":23852,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that AtomProNet, an open-source Python package, can automate the full data flow from crystal-structure retrieval through DFT job submission to neural-network training, and that in an alumina case study the resulting…","keywords":["machine learning interatomic potentials","density functional theory","neural network potential training","equivariant graph neural networks","alumina","open-source software","atomistic simulation workflow","molecular dynamics benchmarking"],"falsifier":"Refit or reparameterize a classical reactive potential on the same 16,000-structure alumina dataset and repeat the property comparison in Table 1; if the refitted classical potential then matches Allegro and MACE on elastic constants and vacancy energies, the claim of MLIP superiority for alumina is falsified. A second decisive check is to evaluate the trained MLIPs on out-of-distribution structures, such as other alumina polymorphs, surfaces, or high-temperature snapshots; if their error rises to classical-potential levels there, the practical-advantage claim is weakened.","tokens_in":16955,"feed_emoji":"⚛️","tokens_out":10660,"duration_ms":88650,"temperature":0.7,"pith_summary":"Guided by a global survey of the materials-science community, the authors present AtomProNet, an open-source Python package that automates the data pipeline behind machine-learning interatomic potentials: retrieving crystal structures from open databases, preparing and submitting density-functional-theory (DFT) jobs, collecting energies and forces, and formatting them for neural-network training. The authors argue this removes a major access barrier for researchers who cannot afford to stitch together many specialized tools. In a case study on alumina, the package generated and processed thousands of strained configurations, and the resulting MLIPs (Allegro and MACE) reproduced DFT elastic constants, cohesive energies, and vacancy formation energies far better than the classical reactive potentials ReaxFF and COMB3. The paper also reports that the MLIPs are computationally practical, with Allegro fastest on a single CPU core across the tested range up to 2.5 million atoms, though the classical potentials parallelize better at very large core counts when charge equilibration is run infrequently. If these claims hold, AtomProNet gives a reproducible route from a material's structure to a working machine-learned interatomic potential.","feed_headline":"AtomProNet automates DFT data flow for machine-learned potentials","feed_subtitle":"The package's alumina case study shows neural-network potentials beat classical force fields on elasticity and defects.","key_machinery":"The machinery is the package's four-module workflow: (a) data collection from open crystallographic databases, (b) data generation through automated DFT job creation and submission, (c) preprocessing that turns raw DFT outputs into neural-network training data, and (d) post-processing that produces parity plots, error distributions, and molecular-dynamics benchmarks. The alumina case study exercises all four modules end-to-end. The trained models are equivariant graph neural networks, meaning their features transform predictably under rotations of the atomic environment; the network outputs atomic energies, and forces are obtained by analytic gradients of those energies, which is the mechanism that lets a few thousand structures generalize to elastic and defect properties.","core_discovery":"The central discovery is that one software layer can carry the full data flow from open crystallographic databases to trained machine-learning interatomic potentials, and that this pipeline produces potentials with clear practical advantages over generic classical potentials for alumina. The authors use AtomProNet to relax an 80-atom alumina supercell containing aluminium and oxygen vacancies, generate roughly 12,000 hydrostatically strained variants (with 16,000 structures total used after data collection), run DFT self-consistent-field calculations on them, and train two equivariant graph-neural-network potentials, Allegro and MACE, on the energies and forces. On held-out validation structures, both models reach $R^2 \\approx 0.99$ for energy, MACE reaches $R^2 = 1$ for forces, and Allegro reaches $R^2 = 0.96$. Compared with DFT references, Allegro and MACE closely track lattice constants, elastic constants, cohesive energy, and vacancy formation energies, whereas ReaxFF and COMB3 overestimate elastic stiffness by hundreds of gigapascals and underestimate binding energies by more than one electronvolt per atom. The authors conclude that machine-learning potentials are both more accurate and practical, with Allegro outrunning the reactive potentials on a single CPU core for systems up to millions of atoms, while the classical potentials win only at very large core counts depending on how often charge equilibration is computed.","pith_inferences":["The accuracy comparison is not apples-to-apples: the MLIPs were trained on the same DFT data used for evaluation, while the ReaxFF and COMB3 parameter sets were generic and not refitted to those data; refitting the classical potentials on the same 16,000 structures could shrink or close the gap.","The paper reports both 12,000 generated strained cases and 16,000 structures used for training; reconciling these counts is a necessary step before treating the dataset-size claim as settled.","A natural extension is active learning: because the package already handles data generation and collection, it could be modified to generate new DFT data for structures where the current model is most uncertain, improving transferability to phases and surfaces outside the training distribution.","The benchmark's hardware-dependence suggests that future comparisons should report cost curves (accuracy versus CPU-hours at fixed system size) rather than a single accuracy table, since the best potential depends on the available computing resources."],"forward_implications":["A researcher with access to density-functional-theory codes can go from a crystal structure to a trained machine-learning potential without writing custom data-handling code.","For ceramics like alumina, machine-learned potentials trained on DFT data can replace classical reactive potentials in simulations of strain, vacancy formation, and mechanical response.","Because the pipeline is scripted, MLIP training and benchmarking become reproducible and easy to share, making comparisons across materials and models more standardized.","The benchmark supplies a practical rule of thumb: on moderate core counts and systems up to millions of atoms, the MLIPs are both more accurate and faster, while classical potentials become competitive only when the job is large enough to amortize their better parallel scaling.","The training set deliberately includes vacancy and strained configurations, so the resulting potentials target defect and spall-failure studies directly."],"supporting_citations":[{"why":"Supplies the crystal structures that the package's data-collection module fetches for the case study.","marker":"[3]"},{"why":"Density-functional-theory code used to relax the alumina supercell and generate energies and forces for the training set.","marker":"[20]"},{"why":"Molecular-dynamics engine in which all four potentials were benchmarked for accuracy and computational performance.","marker":"[22]"},{"why":"Provides the DFT convergence settings (energy cutoff, k-points, smearing) used in the data-generation module.","marker":"[73]"},{"why":"One of the two graph-neural-network architectures trained on the package's output and evaluated against DFT.","marker":"[75]"},{"why":"The other graph-neural-network architecture trained on the package's output; it produced the fastest single-CPU model in the benchmark.","marker":"[76]"},{"why":"Baseline classical reactive potential whose predicted alumina properties are compared with DFT and MLIPs in Table 1.","marker":"[96]"},{"why":"Baseline charge-optimized classical potential whose predicted alumina properties are compared with DFT and MLIPs in Table 1.","marker":"[97]"}],"fun_headline_variants":["Data-to-training pipeline for MLIPs, now automated in AtomProNet","From DFT data to trained neural network potentials, all in one package","AtomProNet: end-to-end automation for MLIP training data","MLIPs outpace classical potentials in alumina test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that comparing MLIPs trained on alumina DFT data with classical potentials that were not fitted to those data is a fair test of which potential family is more accurate; if the classical potentials had been fitted to the same data, or the MLIPs tested on very different structures, the reported accuracy gap could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Data-to-training pipeline for MLIPs, now automated in AtomProNet","From DFT data to trained neural network potentials, all in one package","AtomProNet: end-to-end automation for MLIP training data","MLIPs outpace classical potentials in alumina test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1542,"prompt_tokens":993,"completion_tokens":549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":475}},"tokens_in":609,"tokens_out":549,"duration_ms":5235,"temperature":1.0,"reasoning_tokens":475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:27:18.686303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Refit or reparameterize a classical reactive potential on the same 16,000-structure alumina dataset and repeat the property comparison in Table 1; if the refitted classical potential then matches Allegro and MACE on elastic constants and vacancy energies, the claim of MLIP superiority for alumina is falsified. A second decisive check is to evaluate the trained MLIPs on out-of-distribution structures, such as other alumina polymorphs, surfaces, or high-temperature snapshots; if their error rises to classical-potential levels there, the practical-advantage claim is weakened.","supporting_citations":[{"cited_title":"Fast parallel algorithms for short-range molecular dynamics,","cited_arxiv_id":null,"evidence_quote":"Molecular-dynamics engine in which all four potentials were benchmarked for accuracy and computational performance."},{"cited_title":"Engineering chemo-mechanical properties of zn surfaces via alucone coating,","cited_arxiv_id":null,"evidence_quote":"Provides the DFT convergence settings (energy cutoff, k-points, smearing) used in the data-generation module."},{"cited_title":"Mace: Higher order equivariant message passing neural networks for fast and accurate force fields,","cited_arxiv_id":null,"evidence_quote":"One of the two graph-neural-network architectures trained on the package's output and evaluated against DFT."},{"cited_title":"Learning local equivariant representations for large-scale atomistic dynamics,","cited_arxiv_id":null,"evidence_quote":"The other graph-neural-network architecture trained on the package's output; it produced the fastest single-CPU model in the benchmark."},{"cited_title":"Atomistic-scale analysis of carbon coating and its effect on the oxidation of aluminum nanoparticles by reaxff-molecular dynamics simulations,","cited_arxiv_id":null,"evidence_quote":"Baseline classical reactive potential whose predicted alumina properties are compared with DFT and MLIPs in Table 1."},{"cited_title":"Charge op- timized many-body (COMB) potential for al2o3materials, interfaces, and nanostructures,","cited_arxiv_id":null,"evidence_quote":"Baseline charge-optimized classical potential whose predicted alumina properties are compared with DFT and MLIPs in Table 1."}],"review_version":1}