Pith. sign in

REVIEW 4 major objections 5 minor 12 references

CheMLFlow: An Open-Source Platform for Cheminformatics and Materials Informatics Applications

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CheMLFlow packages complete ML pipelines for chemistry as one reproducible unit and matches published results.

desk verdict Useful platform with an abstract that oversells its benchmark evidence; the software itself is a genuine contribution. read the letter →

arxiv 2608.04942 v1 pith:6A7WY245 submitted 2026-08-05 cs.LG cond-mat.mtrl-scicond-mat.othercs.AIphysics.chem-ph

Brendan Smith , Susana Lopez-Moreno , Eric Dolores-Cuenca , Sangil Kim , Jose L. Mendoza-Cortes , Nijamudheen Abdulrahiman This is my paper · ORCID
classification cs.LGcond-mat.mtrl-scicond-mat.othercs.AIphysics.chem-ph
keywords cheminformaticsmaterialsinformaticsreproduciblemachinelearningworkflowsmolecularpropertypredictiondesignofexperimentsbioactivityagent-assistedscienceopen-sourcesoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CheMLFlow is an open-source platform whose central claim is that a complete scientific machine-learning workflow—from data retrieval and curation through representation, splitting, model training, validation, and reporting—can be packaged as a single reproducible benchmark unit. The paper argues that this reduces the orchestration burden that slows cheminformatics and materials informatics research, and it demonstrates the point by running design-of-experiments grids that match or approach published performance on quantum-mechanical, physicochemical, and bioactivity property prediction (for example, $R^2 = 0.81$ against a published $0.85$ for PDE4B, and $R^2 = 0.69$ against $0.66$ for ALK-5). It also shows the same workflow abstraction extending to an odor-classification task run with a coding agent and to chaotic time-series forecasting. A sympathetic reader would care because the contribution is not a new model or descriptor but a way to make comparisons across complete pipeline configurations auditable and cheap enough to run at scale.

What carries the argument

The load-bearing mechanism is the design-of-experiments (DOE) layer combined with the artifact contract. A DOE specification expands a product grid of models, feature representations, scalers, and split strategies into many single-run configuration files; each case writes its own resolved configuration, status, split metadata, metrics, and predictions to a run directory, and failed cases are recorded as failed rather than converted into artificial numbers. Analysis then aggregates successful child runs into parent-level leaderboard summaries. This makes the complete workflow configuration the reproducible unit of comparison and gives coding agents a structured, inspectable state on which to act.

What would settle it

Rerun one cited ChEMBL study through CheMLFlow using the original paper's exact data version, assay filtering, descriptor set, split definition, and hyperparameter procedure; if the published metric, such as Pred-hERG's $R^2 = 0.61$, cannot be recovered, then the platform adds protocol bias rather than neutral orchestration. The same test applies to the six benchmark datasets: reproduce each cited baseline's exact protocol through CheMLFlow and compare against Table 1, and a systematic shortfall below the cited values would falsify the claim of reaching literature performance.

Watch

Extended reading notes

Core claim

On the paper's own terms, CheMLFlow's discovery is that workflow orchestration can be made the scientific unit of comparison. Every run resolves a YAML/JSON configuration into explicit nodes and writes a standard artifact set—resolved configuration, status, split definitions, metrics, predictions, serialized models, plots, and failure records—so that a benchmark is defined by dataset plus curation plus representation plus split plus preprocessing plus model family, not by a model name alone. Using this protocol, the authors report that single-shot DOE grids reach literature-adjacent performance across six published small-molecule datasets (QM9, P-gp, androgen receptor, flash point, sooting index, PAH) and four ChEMBL bioactivity targets, with random-split results that match or approach the cited baselines and scaffold splits that degrade predictably. They also report that a frozen pretrained CheMeleon encoder and a from-scratch Chemprop network both slot into the same grid, and that the identical abstraction forecasts a chaotic Mackey-Glass series with RMSE values close to the published Adaptive NVAR benchmark.

Load-bearing premise

The central claim rests on treating the platform's single-shot, default-configuration results as comparable to published numbers obtained with different data versions, curation choices, and hyperparameter tuning.

Editorial extensions

If this is right

  • Benchmark tables become regenerable from recorded artifacts, so a reported number can be traced back to one resolved configuration rather than to a model name.
  • Researchers can sweep hundreds or thousands of workflow combinations, as the paper does with 4,080 child jobs, while keeping failures visible instead of silently dropping them.
  • Random splits and scaffold splits can be reported side by side, giving a built-in estimate of how much performance is interpolation versus generalization to new chemical series.
  • The same node-based abstraction extends beyond molecules, as shown with chaotic time-series forecasting, so the platform's benchmarking unit can be reused for other scientific data modalities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair reading is that 'reaches literature performance' is a weaker statement than 'reproduces literature performance': the cited studies used different ChEMBL versions, descriptor sets, and hyperparameter optimization, so the observed gaps (for example, hERG at $R^2 = 0.52$ versus $0.61$) cannot be cleanly separated into platform cost versus protocol difference.
  • If this workflow-unit convention became standard, published state-of-the-art numbers would need to carry their split, curation, and representation metadata to be comparable at all—a cultural shift the paper demonstrates but does not itself enforce.
  • The agent-assisted comparison is a single demonstration with one agent and two task variants; a natural next test is to vary the agent, the prompt, and the seed many times and check whether the artifact contract keeps results stable across those variations.
  • The observed scaffold-split drops of roughly 0.1–0.15 in $R^2$ for three of the four ChEMBL targets suggest a quantitative rule of thumb: models tuned under random splits should be expected to lose about that much when moved to scaffold splits on similar bioactivity data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CheMLFlow, an open-source configuration-driven platform for building end-to-end cheminformatics and materials informatics workflows. The platform layers a workflow builder, a design-of-experiments (DOE) layer, and an agent-facing skill layer, with an artifact contract that records resolved configurations, split definitions, metrics, predictions, and failures for each run. The authors demonstrate the platform on ChEMBL-derived bioactivity prediction tasks, six molecular property benchmark datasets (QM9, Pgp, ARA, Flash, YSI, PAH), an agent-assisted odor-prediction study on QuantumScents, and a time-series forecasting module using the Adaptive NVAR model on Mackey-Glass data. The central claim is that the platform's benchmarks 'reach literature performance' for quantum mechanical, physicochemical, and bioactivity property prediction while making workflows explicit, repeatable, comparable, and inspectable.

Significance. If the claims are substantiated, CheMLFlow addresses a real bottleneck: the orchestration burden of assembling data acquisition, curation, representation, training, validation, screening, and reporting into reproducible pipelines. The artifact contract, DOE aggregation that retains failed runs, and explicit treatment of split strategies are genuine strengths. The platform is open source, and the agent-assisted workflow evaluation is a useful demonstration of human-on-the-loop automation. The paper is therefore potentially valuable to the cheminformatics and materials informatics communities. However, the headline claim of 'reaching literature performance' is not fully supported by the evidence as presented, because the comparisons are explicitly disavowed as exact reproductions, the data are live and unpinned, and the DOE runs are single-shot without hyperparameter optimization.

major comments (4)
  1. [Abstract; Section 3.1; Section 3.2] The abstract and conclusions state that CheMLFlow benchmarks 'reach literature performance,' but Section 3.1 explicitly states that 'These comparisons were not intended as exact reproductions' and instead describes the results as 'literature adjacent' and 'literature comparable.' Section 3.2 similarly claims that the DOE approach 'matched the performance metrics of literature SOTA methods,' yet Table 1 shows differences such as QM9 MAE 0.004736 versus 0.0043 and YSI MAE 22.34 versus 19.17. Because the central claim of the paper is that the platform reaches literature performance, the abstract and conclusions should be aligned with the actually supported claim: that the platform produces literature-comparable baselines under a transparent protocol.
  2. [Section 3.1; Section 3.2; Section 2.2] The comparability of the numerical benchmarks to literature values is not established. The ChEMBL targets are retrieved live with no pinned database version, the curation and split definitions follow CheMLFlow defaults rather than the cited papers, and Section 3.2 states that Optuna hyperparameter optimization was 'not attempted here.' Many literature values come from tuned or specialized pipelines, so the observed deltas cannot be attributed to the platform. The paper should supply, at minimum, the exact data versions, assay filtering criteria, split definitions, and hyperparameter settings for each benchmark, ideally as archived run artifacts with commit hashes, so that readers can judge whether the comparisons are meaningful.
  3. [Section 3.4; Supporting Information Table S1] The time-series benchmark validates CheMLFlow's Adaptive NVAR implementation against the authors' own previous paper (Ref. 47, with overlapping authors). This is a self-consistency check rather than an independent literature comparison, and the Supporting Information table appears to lack clear column headers and error bars or standard deviations. Please either add an independent benchmark or explicitly label this as a reproducibility check against the original implementation, and report the distribution of results across repeats.
  4. [Section 2.2; Section 5] The reproducibility claim is not fully verifiable because the paper does not provide a pinned commit hash, an environment lock, or archived benchmark run artifacts. Section 2.2 mentions that runs record a 'git hash,' but the paper's Software and Data Availability section lists only a GitHub URL. To support the claim that the reported tables can be regenerated, the authors should include the exact repository version, dependency versions, and the resolved configuration hashes for all benchmark runs in the supplementary material or an archived repository.
minor comments (5)
  1. [Section 3.2, Table 1] The column header 'Performance Reported CheMLFlow (single shot DOE run) This study Literature' is difficult to parse; please separate the 'This study' and 'Literature' columns with clear headings and indicate which metric each column reports.
  2. [Section 3.3] The comparison between the standalone agent result and the CheMLFlow result is confounded by different validation strategies (stratified random versus scaffold splits); the text should explicitly acknowledge that the 0.88 ROC-AUC values are not directly comparable across the two protocols.
  3. [Section 3.3] The agent experiments rely on a proprietary model ('Codex powered by GPT-5.5') without a version date or release identifier; please specify the exact model version and snapshot so that the experiment is repeatable.
  4. [Supporting Information, Table S1] The table header 'Perf (RMSE) SOTA Perf' appears truncated, and the two numerical columns are not clearly labeled as 'CheMLFlow' and 'Reference'; please format the table with complete headers.
  5. [Section 3.1] The phrase 'literature adjacent' is informal for a journal article; consider using 'literature-comparable' throughout, and ensure that the abstract uses the same qualified language.

Circularity Check

1 steps flagged · score 4.0 of 10

Time-series benchmark leans on the authors' own prior Adaptive NVAR paper, while the core six-dataset and ChEMBL benchmarks are external and non-circular.

  1. self citation load bearing [Section 3.4 (Time Series Forecasting) and Table S1]
    "CheMLFlow’s implementation of a reported benchmark gives the results in Table S1. ... The results are then compared with those originally reported in the Adaptive NV AR paper, showing that CheMLFlow successfully forecasts this chaotic time series with performance comparable to the reported benchmark."

    'SOTA Perf' in Table S1 is the performance reported in Ref. 47, authored by López-Moreno, Dolores-Cuenca, and Kim, who are co-authors of the present paper. The comparison that supports the statement 'CheMLFlow successfully forecasts ... comparable to the reported benchmark' is therefore CheMLFlow against the same group's own prior Adaptive NVAR numbers, not against an external standard. The model, the benchmark task, and the reference values all originate from the same group, so the time-series demonstration reduces to a self-consistency check: the platform is being validated by restating the authors' own earlier results as the 'reported benchmark.' This does not corrupt the external six-dataset benchmarks, but it is load-bearing for the abstract's 'beyond molecular' use-case claim.

full rationale

Most of the paper's central benchmark chain is not circular. Table 1 compares CheMLFlow's single-shot DOE runs on QM9, Pgp, ARA, Flash, YSI, and PAH against literature values from independent external papers; the runs use fixed public datasets, standard featurizations, and cross-validation splits, and the paper explicitly says Optuna hyperparameter optimization 'is not attempted here' (Section 3.2). Those are ordinary benchmark comparisons, not fitted inputs relabeled as predictions. Similarly, the ChEMBL bioactivity section compares against external papers (Refs 28-31) and immediately disclaims exact reproduction: 'These comparisons were not intended as exact reproductions ... ChEMBL version, assay filtering, descriptor choices, hyperparameter optimization, and split definitions can all affect final performance' (Section 3.1). That disclaimer weakens the abstract's phrase 'reaches literature performance' as a comparability matter, but it is not circularity. The only self-referential element that carries real weight is the time-series evaluation, where the comparison target is the same group's own Adaptive NVAR paper (Ref. 47). Since that is a benchmark against the authors' own prior numbers, it should be read as a reproducibility check rather than independent confirmation. The agent experiment uses CheMLFlow to evaluate CheMLFlow's agent skills, but the outcome metric is an external odor-classification ROC-AUC on QuantumScents, so it is not circular by construction. Overall, one load-bearing-for-ancillary-claim self-citation exists while the main property-prediction claims rest on external benchmarks, giving a score of 4.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No parameters were fitted to data in this paper; the benchmark models use defaults, and the central claim is about the platform, not a predictive model. The axioms are standard domain assumptions plus the self-referential Adaptive NVAR fidelity assumption. No new physical or conceptual entities are introduced; CheMLFlow is software, not a new scientific object.

assumptions (5)
  • domain assumption ChEMBL IC50 records, after curation and pIC50 conversion, provide a valid endpoint for the bioactivity benchmarks.
    QSAR modeling relies on the assumption that publicly curated bioactivity data are reliable enough for comparative benchmarking (Section 3.1).
  • domain assumption Scaffold splits give a conservative estimate of generalization to new chemical series, and random splits estimate interpolation performance.
    This is a standard assumption in cheminformatics, stated in Section 3.1, and used to interpret the benchmark results.
  • domain assumption The six small-molecule datasets (QM9, Pgp, ARA, Flash, YSI, PAH) are representative benchmarks for model comparison.
    The datasets were taken from a recent study by Burns and Green (Ref 42), and the paper assumes they allow direct comparison with literature performance (Section 3.2).
  • domain assumption Rerunning the same resolved configuration with the same random seed and compatible software environment reproduces the same pipeline structure and split definitions.
    This reproducibility premise is central to the artifact contract (Section 2.2) and depends on deterministic behavior of all components, which is not guaranteed across software versions.
  • ad hoc to paper The Adaptive NVAR model implementation in CheMLFlow is faithful to the original published model.
    The benchmark compares CheMLFlow's implementation to results in the authors' own prior paper (Ref 47, Section 3.4 and Table S1), assuming the implementation matches the original.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CheMLFlow: An Open-Source Platform for Cheminformatics and Materials Informatics Applications." pith.science (2026). https://pith.science/paper/6A7WY245

@misc{pith2026260804942,
  author       = {Pith},
  title        = {Pith review of: CheMLFlow: An Open-Source Platform for Cheminformatics and Materials Informatics Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6A7WY245}},
  note         = {Machine review of arXiv:2608.04942}
}
read the original abstract

CheMLFlow is an open-source platform for building and executing end-to-end, high-throughput, and agentic workflows for scientific and technological applications. CheMLFlow targets a common bottleneck in scientific machine learning development, where researchers often need to assemble data acquisition, curation, representation, model training, validation, screening, interpretation, and reporting into a reproducible pipeline, even when their primary research contribution concerns only one stage. CheMLFlow provides modular workflow components, ready-to-run reference pipelines, standardized artifacts, and evaluation outputs that reduce orchestration overhead and support benchmarking across methods and datasets. The platform is designed to be extensible, reproducible, and automation friendly, with pluggable representations and models, deterministic splits, explicit run artifacts, batch execution, and report generation. As scientific software increasingly moves toward agent assisted experimentation, CheMLFlow's configuration driven workflows and structured outputs also provide a practical interface for coding agents to help users construct experiments, inspect results, and summarize findings under human supervision. This article describes the system architecture, core workflows, and benchmarks that reach literature performance for quantum mechanical, physicochemical and bioactivity property prediction, and use cases involving time series datasets demonstrating applications beyond molecular chemistry datasets.

Figures

Figures reproduced from arXiv: 2608.04942 by the authors.

Figure 2
Figure 2. CheMLFlow tree structure presenting definitions of DOE guided high throughput runs of model combinations and individual config driven workflow runs along with options available at individual nodes. The scripts and files defining options at node level are also shown. DOE specification config/*doe*.yaml experiment grid: models, features, splits, defaults Manual runtime configuration config/*.yaml or tutorial config on… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

  1. [1]

    INTRODUCTION Machine learning (ML) and artificial intelligence (AI) are reshaping scientific and technological research by providing practical tools to understand, predict, and design complex molecular and material systems. Cheminformatics and materials informatics combine ML/AI with traditional computational chemistry, statistical modeling, and domain sp...

    work page

  2. [2]

    SOFTWARE AND METHODS 2.1. System Design and Workflow Specification CheMLFlow is designed as an open-source platform to implement ML/AI workflows (Figure 1) using a command line interface (CLI) and/or an agent. A workflow is designed as Data generation/ acquisition ● External DB ● Local DB ● Benchmarks ● Schemas ● Clean ● De-duplicate ● Standardise ● Label...

    work page

  3. [3]

    APPLICATIONS AND RESULTS 3.1. Application of CheMLFlow to Bioactivity Prediction We demonstrate the use of CheMLFlow for DOE-driven, end-to-end ML/AI training for bioactivity (IC50) prediction tasks using datasets retrieved from ChEMBL database in the initial step of the workflow run.27 In each study, DOE specification was defined by a ChEMBL target prote...

    work page

  4. [4]

    This work demonstrated CheMLFlow as a chemistry aware, agent-assisted ML/AI workflow development and experimentation platform

    CONCLUSIONS We have developed CheMLFlow, an open-source, configuration driven software for building, benchmarking, and applying reproducible and explainable molecular ML/AI workflows in a high throughput fashion. This work demonstrated CheMLFlow as a chemistry aware, agent-assisted ML/AI workflow development and experimentation platform. CheMLFlow was app...

    work page

  5. [5]

    SOFTWARE AND DATA A V AILABILITY CheMLFlow is available at https://github.com/nijamudheen/CheMLFlow

    work page

  6. [6]

    developed the original idea

    AUTHOR CONTRIBUTIONS N.A. developed the original idea. B.S., S.L-M., E.D-C., and N.A. conceptualized the ideas, implemented the software, designed and run the experiments, and analyzed the results. B.S. and N.A. wrote the manuscript with contributions from S.L.M. and E.D.C. S.K. supervised the implementation of Adaptive NV AR model. All authors have read ...

    work page

  7. [8]

    Figure 2

    Analysis outputs such as aggregate CSV/JSON summaries and figures generated from DOE or run artifacts where the analysis workflow is executed. Figure 2. CheMLFlow tree structure presenting definitions of DOE guided high throughput runs of model combinations and individual config driven workflow runs along with options available at individual nodes. The sc...

    work page

  8. [9]

    We are grateful for the computational support from the Institute for Cyber-Enabled Research (ICER) at the Michigan State University

    ACKNOWLEDGEMENTS This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. We are grateful for the computational support from the Institute for Cyber-Enabled Research (ICER) at the Michigan State University. S.L.M., E.D.C., and S.K. are grateful for the support from the National Research Foundat...

    work page 2024

Show all 12 references
  1. [10]

    Mendoza-Cortes: https://orcid.org/0000-0001-5184-1406 Nijamudheen Abdulrahiman: https://orcid.org/0000-0001-9191-1851

    ORCID Brendan Smith: https://orcid.org/0000-0003-3460-9984 Susana López-Moreno: https://orcid.org/0000-0002-1322-2775 Eric Dolores-Cuenca: https://orcid.org/0000-0003-3909-5580 Sangil Kim: https://orcid.org/0000-0002-4408-2904 Jose L. Mendoza-Cortes: https://orcid.org/0000-000...

  2. [11]

    Mendoza-Cortes: jmendoza@msu.edu Nijamudheen Abdulrahiman: apchnijam@gmail.com, nijam@kernfieldlabs.com 26

    CORRESPONDING AUTHOR Jose L. Mendoza-Cortes: jmendoza@msu.edu Nijamudheen Abdulrahiman: apchnijam@gmail.com, nijam@kernfieldlabs.com 26

  3. [12]

    REFERENCES (1) Seal, S.; Mahale, M.; García-Ortegón, M.; Joshi, C. K.; Hosseini-Gerami, L.; Beatson, A.; Greenig, M.; Shekhar, M.; Patra, A.; Weis, C.; Mehrjou, A.; Badré, A.; Paisley, B.; Lowe, R.; Singh, S.; Shah, F.; Johannesson, B.; Williams, D.; Rouquie, D.; Clevert, D.-A...

    2025

  4. [13]

    The model architecture is shown in Figure S1

    Adaptive NVAR Model Implementation Adaptive NVAR is a machine-learning algorithm for time-series forecasting with a simple architecture that improves scalability and robustness to noisy data. The model architecture is shown in Figure S1. The benchmarking results are summarized...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.