REVIEW 6 minor 3 references
Best Practices for Machine Learning-Assisted Protein Engineering
T0 review · 0 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that supervised machine learning for protein engineering is fundamentally a software engineering problem, and that disciplined data handling, validation, and code practices—not novel algorithms—are what make ML models…
desk verdict A solid, clearly written perspective that consolidates existing ML best practices for protein engineering; the value is in the synthesis and the PECC repository, not in new results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The guiding mechanism is the supervised learning pipeline treated as a software artifact. The load-bearing components are: evolutionary-aware data splitting to prevent leakage; log-transformed labels so improvements and losses are symmetric around wild type; a strict train-validation-test separation with k-fold cross-validation for small datasets; and modular, version-controlled, documented code. The paper's named deliverable, the Protein Engineering Code Center, operationalizes these rules as reusable tutorials and scripts. Together these practices are what the paper claims convert an ML project from an irreproducible experiment into a dependable engineering process.
What would settle it
Compare two pipelines on the same protein engineering dataset with identical experimental labels: one following the paper's full checklist and one using random splits and unversioned code. If the poorly engineered model matches the well-engineered one on evolutionarily held-out test sets across several datasets, the claim that process determines reliability is weakened. Alternatively, a systematic survey showing that most reported failures trace to label noise or insufficient data, not to split leakage or code errors, would falsify the paper's emphasis.
Extended reading notes
Core claim
The central claim is that the reliability of ML-assisted protein engineering is determined less by the choice of algorithm than by the rigor of the surrounding workflow. The paper asserts that ML development is a form of software engineering, and accordingly that the field's recurring failures—overfitting, data leakage, irreproducibility—are symptoms of weak process. Its prescription is a concrete checklist: clean and normalize labels (with log transformation for enzyme variant data), split data with awareness of evolutionary relationships and epistasis, validate with k-fold cross-validation plus a held-out test set, modularize code, version-control data and models, and deploy with documentation. The deliverable is both a written roadmap and a companion code center meant to make these practices adoptable.
Load-bearing premise
The load-bearing premise is that weak software engineering practice—more than data scarcity, experimental noise, or inherently hard prediction problems—is the main cause of unreliable machine learning in protein engineering; if the dominant failures are elsewhere, the checklist would miss the core issue.
Editorial extensions
If this is right
- If followed, the guidelines should make ML results in protein engineering reproducible across labs, because data splits, preprocessing, and code are specified and versioned.
- Journals and editors can use the checklist as a review standard, catching leakage or missing test sets before publication.
- Practitioners can compare models fairly across studies, since evaluation protocols (same split logic, same metrics) become consistent.
- Newcomers can enter the field with a clear path, lowering the barrier posed by the daily flood of new methods.
- Well-structured models reduce debugging time and support iterative design cycles, making wet-lab validation more informative.
Reading between the lines
- The paper's premise could be tested empirically: run the same prediction task twice, once following the checklist and once not, and compare generalization on evolutionarily distant variants; the authors do not report such a benchmark.
- The same software-engineering discipline likely extends beyond supervised regression to generative models and zero-shot predictors, where data splitting and validation are even less standardized.
- If journals adopted the checklist as a requirement, it would create pressure for shared baselines and might accelerate adoption of standardized benchmark datasets in protein engineering.
- A quantitative failure-mode survey of recent ML protein engineering papers—counting how many use random splits or lack a holdout—would show how much of the problem is actually process-related.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript is a perspective article that argues for treating machine learning (ML) development in protein engineering as a software engineering discipline. It provides a step-by-step set of best practices for supervised ML workflows, covering experimental data collection and cleaning, label normalization, exploratory analysis, data splitting with attention to evolutionary relationships, protein sequence featurization, dimensionality reduction, algorithm selection, hyperparameter tuning, model evaluation, code quality, and deployment. It also introduces the Protein Engineering Code Center (PECC), a public GitHub repository with tutorials and code examples implementing the suggested workflows. The central claim is that 'better modeling practices result in better outcomes' and that following these guidelines will make ML-assisted protein engineering more reproducible and reliable.
Significance. The paper is a valuable synthesis of existing community recommendations (e.g., DOME, REFORMS) tailored to protein engineering. Its strengths are the explicit treatment of data leakage through evolutionary-aware splits, the emphasis on software engineering practices such as version control and modular code, and the practical PECC resource. While the paper presents no new empirical evidence that these practices improve outcomes, that is appropriate for a perspective; the 'better outcomes' claim is a normative guideline. The recommendations are generally accurate and well-referenced, and the paper is clearly written. The main weaknesses are a few overly prescriptive technical suggestions that need caveats, but they do not undermine the central argument.
minor comments (6)
- [1.2] The recommendation to apply a logarithmic transformation to enzyme variant labels (Section 1.2) should explicitly address cases where labels are not strictly positive, such as ΔΔG, melting temperature in °C, or measurements that include zero or negative values. A log transform is undefined or not meaningful for such data, and the current phrasing could mislead practitioners; a caveat or alternative (e.g., a signed transform) would improve the guidance.
- [4] The mention of McNemar's test in Section 4 should clarify that it is appropriate only for paired categorical outcomes (e.g., classification) and not for regression metrics like RMSE or MAE, which are the focus of the preceding paragraph. Please specify the context or add a note about when McNemar's test applies.
- [1.4] The suggested 70:10:20 training/validation/test split and the default of 10-fold cross-validation are reasonable but are presented without empirical justification. A brief note that these choices depend on dataset size and task type would make the guidance more robust.
- [6] The PECC repository is presented as a central resource. Please include a version or accession date and, ideally, a permanent identifier (e.g., a Zenodo DOI) to ensure long-term accessibility; the current reference to a GitHub repository alone may not be stable.
- [2.1] The text references 'iFeature65' but Reference 65 is missing the tool name in its title; it should read 'iFeature: a Python package and web server for features extraction and selection from protein and peptide sequences.'
- [Fig. 5] In the caption of Figure 5, 'Hu gging Face' appears to be a typo; it should be 'Hugging Face'.
Circularity Check
No significant circularity: the perspective's recommendations rest on external consensus, and its self-citations are background, not load-bearing.
full rationale
This is a perspectives and guidelines paper, not an empirical study: it contains no fitted parameters, no benchmark predictions, and no derivation chain that could reduce to its own inputs. The central claim ('better modeling practices result in better outcomes,' Conclusion) is a normative recommendation, and the paper explicitly disclaims novelty in techniques, positioning itself as a compilation of quality standards. The recommendations on data splitting, validation, and code quality are grounded in external literature (DOME, ref. 36; REFORMS, ref. 37; Bernett et al. on data leakage, ref. 51), and the software-engineering premise (Sec. 1, p. 4: 'ML projects in protein often overlook the critical aspect that ML development is fundamentally a form of software engineering') is cited to external software-engineering studies (refs. 28-29), not to the authors' own work. The paper does contain several self-citations (refs. 6, 15, 22, 56 to Davari et al. and Cadet et al.), but these support background statements, such as that data-driven approaches are becoming central to protein engineering and that epistatic interactions matter; none of these citations carries the weight of the central argument, and no uniqueness theorem or fitted value is invoked. No equation in the paper equals its inputs by construction, and no fitted quantity is renamed as a prediction. The asserted but untested premise that weak software-engineering practice is a principal cause of unreliable ML is a robustness concern, not a circularity. The paper is self-contained against external consensus and deserves a low circularity score.
Assumptions & free parameters
assumptions (4)
- domain assumption Supervised learning is the most relevant ML paradigm for protein engineering.
- ad hoc to paper ML development is fundamentally a form of software engineering.
- domain assumption Random data splitting leads to data leakage and optimistic performance.
- ad hoc to paper Logarithmic transformation of enzyme variant labels is advisable.
Cite this review
Pith. "Pith review of Best Practices for Machine Learning-Assisted Protein Engineering." pith.science (2026). https://pith.science/paper/TEII2DSK
@misc{pith2026250707547,
author = {Pith},
title = {Pith review of: Best Practices for Machine Learning-Assisted Protein Engineering},
year = {2026},
howpublished = {\url{https://pith.science/paper/TEII2DSK}},
note = {Machine review of arXiv:2507.07547}
}
read the original abstract
Data-driven modeling based on Machine Learning (ML) is becoming a central component of protein engineering workflows. This perspective presents the elements necessary to develop effective, reliable, and reproducible ML models, and a set of guidelines for ML developments for protein engineering. This includes a critical discussion of software engineering good practices for development and evaluation of ML-based protein engineering projects, emphasizing supervised learning. These guidelines cover all the necessary steps for ML development, from data acquisition to model deployment. Additionally, the present perspective provides practical resources for the implementation of the outlined guidelines. These recommendations are also intended to support editors and scientific journals in enforcing good practices in ML-based protein engineering publications, promoting high standards across the community. With this, the aim is to further contribute to improved ML transparency and credibility by easing the adoption of software engineering best practices into ML development for protein engineering. We envision that the wide adoption and continuous update of best practices will encourage informed use of ML on real-world problems related to protein engineering.
Figures
Reference graph
Works this paper leans on
-
[1]
1 Paul, C. et al. Enzyme engineering for biocatalysis. Mol Catal 555, doi:10.1016/j.mcat.2024.113874 (2024). 2 Chen, K. & Arnold, F. H. Engineering new catalytic activities in enzymes. Nat Catal 3, 203-213, doi:10.1038/s41929-019-0385-5 (2020). 3 Arnold, F. H. Innovation by Evolution: Bringing New Chemistry to Life (Nobel Lecture). Angew Chem Int Edit 58,...
-
[3]
Nature 630, doi:10.1038/s41586-024- 07487-w (2024). 12 Baek, M. et al. Accurate prediction of protein structures and interactions using a three -track neural network. Science 373, 871 -+, doi:10.1126/science.abj8754 (2021). 13 Yang, J., Li, F. Z. & Arnold, F. H. Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering. Acs Central Sci...
arXiv 2024
-
[2024]
(2024). 35 Heil, B. J. et al. Reproducibility standards for machine learning in the life sciences. Nat Methods 18, 1132-1135, doi:10.1038/s41592- 021-01256-7 (2021). 36 I. Walsh, D. F., D. Garcia-Gasulla, T. Titma, G. Pollastri, J. Harrow, F. E. Psomopoulos and S. C. E. Tosatto. DOME: recommendations for supervised machine learning validation in biology. ...
arXiv 2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.