{"id":"a0d20a99-b502-44d1-9048-0ecc7499a325","arxiv_id":"2507.07547","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A perspective paper that presents a structured set of best practices, with a companion code repository, for supervised machine learning in protein engineering.","lead":"A team of enzyme engineers and data scientists lays out best practices for building machine learning models in protein engineering, covering data, splitting, modeling, evaluation, and deployment. The practices come with a public GitHub repository of tutorials and code to help labs adopt them.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified — the perspective is appropriately scoped; the software-engineering premise is an assertion but not load-bearing for an ACCEPT verdict.","rationale":"The reader's verdict of ACCEPT is appropriate for this perspective paper. The reader's weakest_assumption (that software engineering practice is a principal bottleneck) is fair as a description of what the paper asserts, but it does not constitute a load-bearing weakness: the guidelines cover data handling, modeling, and evaluation comprehensively, and the paper does not claim to have empirically ranked the causes of unreliability. The central claim is normative and self-consistent; the paper explicitly says it does 'not aim to provide the latest cutting-edge techniques but rather to establish quality standards.' Thus, requiring empirical validation of the premise would misapply a hypothesis-testing standard to a best-practices perspective. I found no internal inconsistencies that threaten the core argument. The log-transform advice in Section 1.2 is overgeneralized (absolute labels, zero/negative values), and the McNemar test mention in Section 4 is incomplete without specifying classification, but these are minor refinements. The PECC repository is a concrete, checkable deliverable; verifying it runs would confirm the resource claim, but even a negative result would not invalidate the written guidance. Therefore, no change to the verdict is warranted.","tokens_in":13845,"tokens_out":4034,"duration_ms":44976,"concrete_test":"Clone the Protein Engineering Code Center repository (https://github.com/FabioHerrera97/Protein_Engineering_Code_Center), create a fresh conda environment from the provided environment.yml or requirements.txt, and execute all tutorials end-to-end. Verify that every notebook/script runs without errors and produces the outputs described in the documentation. If the repository is incomplete or non-functional, the paper's practical-resource contribution is overstated, but the written guidelines remain valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. The paper is a perspective, not an empirical study, and its central claim ('better modeling practices result in better outcomes') is a normative guideline rather than a tested causal hypothesis. The reader's weakest assumption — that weak software engineering practice is a principal cause of unreliable ML in protein engineering — is indeed asserted rather than demonstrated, but it is not load-bearing for the paper's acceptance. The recommendations span data collection, cleaning, splitting, representation, model training, evaluation, and deployment (Sections 1–5), not only software engineering, so even if data quality or model expressiveness were the dominant bottleneck, most of the checklist would still address core issues. The paper explicitly scopes itself as a guide rather than a source of novel empirical evidence, and its recommendations align with existing community consensus (e.g., DOME, REFORMS). Minor technical caveats exist — for example, Section 1.2 recommends log-transforming labels without discussing cases where labels are absolute measurements (e.g., melting temperature in °C) or where zero/negative values occur, and Section 4 mentions McNemar's test without restricting it to classification tasks — but these are localized advice details, not threats to the central argument. The practical resource (PECC repository) is claimed to be public and functional; this is worth verifying, but the conceptual guidelines stand independently of the repository's current state.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a perspective article that argues for treating machine learning (ML) development in protein engineering as a software engineering discipline. It provides a step-by-step set of best practices for supervised ML workflows, covering experimental data collection and cleaning, label normalization, exploratory analysis, data splitting with attention to evolutionary relationships, protein sequence featurization, dimensionality reduction, algorithm selection, hyperparameter tuning, model evaluation, code quality, and deployment. It also introduces the Protein Engineering Code Center (PECC), a public GitHub repository with tutorials and code examples implementing the suggested workflows. The central claim is that 'better modeling practices result in better outcomes' and that following these guidelines will make ML-assisted protein engineering more reproducible and reliable.","tokens_in":14040,"tokens_out":6648,"duration_ms":67239,"significance":"The paper is a valuable synthesis of existing community recommendations (e.g., DOME, REFORMS) tailored to protein engineering. Its strengths are the explicit treatment of data leakage through evolutionary-aware splits, the emphasis on software engineering practices such as version control and modular code, and the practical PECC resource. While the paper presents no new empirical evidence that these practices improve outcomes, that is appropriate for a perspective; the 'better outcomes' claim is a normative guideline. The recommendations are generally accurate and well-referenced, and the paper is clearly written. The main weaknesses are a few overly prescriptive technical suggestions that need caveats, but they do not undermine the central argument.","major_comments":[],"minor_comments":[{"comment":"The recommendation to apply a logarithmic transformation to enzyme variant labels (Section 1.2) should explicitly address cases where labels are not strictly positive, such as ΔΔG, melting temperature in °C, or measurements that include zero or negative values. A log transform is undefined or not meaningful for such data, and the current phrasing could mislead practitioners; a caveat or alternative (e.g., a signed transform) would improve the guidance.","section":"1.2"},{"comment":"The mention of McNemar's test in Section 4 should clarify that it is appropriate only for paired categorical outcomes (e.g., classification) and not for regression metrics like RMSE or MAE, which are the focus of the preceding paragraph. Please specify the context or add a note about when McNemar's test applies.","section":"4"},{"comment":"The suggested 70:10:20 training/validation/test split and the default of 10-fold cross-validation are reasonable but are presented without empirical justification. A brief note that these choices depend on dataset size and task type would make the guidance more robust.","section":"1.4"},{"comment":"The PECC repository is presented as a central resource. Please include a version or accession date and, ideally, a permanent identifier (e.g., a Zenodo DOI) to ensure long-term accessibility; the current reference to a GitHub repository alone may not be stable.","section":"6"},{"comment":"The text references 'iFeature65' but Reference 65 is missing the tool name in its title; it should read 'iFeature: a Python package and web server for features extraction and selection from protein and peptide sequences.'","section":"2.1"},{"comment":"In the caption of Figure 5, 'Hu gging Face' appears to be a typo; it should be 'Hugging Face'.","section":"Fig. 5"}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is a perspective and fits the journal's scope. It is not an empirical study, so the lack of data is not a concern. I recommend that the editors or authors verify that the PECC repository is publicly accessible and contains the claimed materials. The self-citations to the authors' previous work (e.g., PyPEF) are used as background context and do not compromise the recommendations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You can take this one at face value: it's a well-scoped perspective that collects current best practices for supervised ML in protein engineering into one place, and it does that job carefully. Nothing here is conceptually new—most recommendations trace to DOME, REFORMS, Heil, Artrith, Bernett, and similar sources—but the protein-engineering framing is useful, and the companion PECC repository gives practitioners something to actually run. That alone justifies a read.\n\nWhat it does well: the pipeline structure is sensible (data collection, cleaning, representation, splitting, training, evaluation, deployment), the advice is consistent with community consensus, and the references are accurate and current. The sections on evolutionary-aware splitting and on code quality are the most valuable; they translate general software engineering norms to a field where many practitioners still split randomly. I also appreciated the explicit statement that this is not a survey of cutting-edge methods—it sets expectations honestly.\n\nSoft spots: the paper asserts rather than shows that weak software engineering is a principal cause of unreliable models. That may be true, but it's not tested here. It doesn't matter much for an ACCEPT, because the checklist is useful even if the bottleneck is elsewhere. Two minor technical caveats: the log-transform advice in Section 1.2 presumes ratio-scale labels; absolute measurements like melting temperature in °C don't obviously benefit, and zero/negative values are a problem. And McNemar's test in Section 4 is only appropriate for classification, not regression—the text should say so. These are details, not flaws in the central argument.\n\nThe self-citations (PyPEF, epistasis work) are appropriate background, not load-bearing. Circularity burden is minimal.\n\nWho this is for: anyone starting an ML-assisted protein engineering project, reviewers who want a rubric, and journals looking for reporting standards. It deserves a serious referee. I'd accept it, and I'd probably read the PECC repository before final sign-off to confirm it matches the paper's claims.\n\nRecommendation: send to peer review. It's a synthesis, but a good one.","headline":"A solid, clearly written perspective that consolidates existing ML best practices for protein engineering; the value is in the synthesis and the PECC repository, not in new results.","tokens_in":14616,"tokens_out":2063,"would_cite":true,"duration_ms":21017,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that supervised machine learning for protein engineering is fundamentally a software engineering problem, and that disciplined data handling, validation, and code practices—not novel algorithms—are what make ML models…","keywords":["protein engineering","machine learning","best practices","supervised learning","software engineering","reproducibility","data splitting","model validation"],"falsifier":"Compare two pipelines on the same protein engineering dataset with identical experimental labels: one following the paper's full checklist and one using random splits and unversioned code. If the poorly engineered model matches the well-engineered one on evolutionarily held-out test sets across several datasets, the claim that process determines reliability is weakened. Alternatively, a systematic survey showing that most reported failures trace to label noise or insufficient data, not to split leakage or code errors, would falsify the paper's emphasis.","tokens_in":13606,"feed_emoji":"🧬","tokens_out":4680,"duration_ms":47019,"temperature":0.7,"pith_summary":"This perspective argues that supervised machine learning for protein engineering is fundamentally a software engineering task, and that the field's reproducibility problems are best fixed by adopting disciplined development practices. It sets out a step-by-step guideline covering data collection, cleaning, log-transformation of labels, evolutionary-aware data splitting, model training and validation, code quality, and deployment. The authors contend that better modeling practices directly produce better outcomes, and they package the recommendations with an open code repository so that newcomers can follow them in practice. A sympathetic reader would take the paper as proposing a community-wide quality standard, not a new algorithm.","feed_headline":"ML protein engineering needs software engineering discipline","feed_subtitle":"A perspective argues that reproducible protein ML comes from disciplined data handling and code, not new algorithms.","key_machinery":"The guiding mechanism is the supervised learning pipeline treated as a software artifact. The load-bearing components are: evolutionary-aware data splitting to prevent leakage; log-transformed labels so improvements and losses are symmetric around wild type; a strict train-validation-test separation with k-fold cross-validation for small datasets; and modular, version-controlled, documented code. The paper's named deliverable, the Protein Engineering Code Center, operationalizes these rules as reusable tutorials and scripts. Together these practices are what the paper claims convert an ML project from an irreproducible experiment into a dependable engineering process.","core_discovery":"The central claim is that the reliability of ML-assisted protein engineering is determined less by the choice of algorithm than by the rigor of the surrounding workflow. The paper asserts that ML development is a form of software engineering, and accordingly that the field's recurring failures—overfitting, data leakage, irreproducibility—are symptoms of weak process. Its prescription is a concrete checklist: clean and normalize labels (with log transformation for enzyme variant data), split data with awareness of evolutionary relationships and epistasis, validate with k-fold cross-validation plus a held-out test set, modularize code, version-control data and models, and deploy with documentation. The deliverable is both a written roadmap and a companion code center meant to make these practices adoptable.","pith_inferences":["The paper's premise could be tested empirically: run the same prediction task twice, once following the checklist and once not, and compare generalization on evolutionarily distant variants; the authors do not report such a benchmark.","The same software-engineering discipline likely extends beyond supervised regression to generative models and zero-shot predictors, where data splitting and validation are even less standardized.","If journals adopted the checklist as a requirement, it would create pressure for shared baselines and might accelerate adoption of standardized benchmark datasets in protein engineering.","A quantitative failure-mode survey of recent ML protein engineering papers—counting how many use random splits or lack a holdout—would show how much of the problem is actually process-related."],"forward_implications":["If followed, the guidelines should make ML results in protein engineering reproducible across labs, because data splits, preprocessing, and code are specified and versioned.","Journals and editors can use the checklist as a review standard, catching leakage or missing test sets before publication.","Practitioners can compare models fairly across studies, since evaluation protocols (same split logic, same metrics) become consistent.","Newcomers can enter the field with a clear path, lowering the barrier posed by the daily flood of new methods.","Well-structured models reduce debugging time and support iterative design cycles, making wet-lab validation more informative."],"supporting_citations":[{"why":"Supplies the software-engineering framing that ML development is a form of software engineering where best practices matter.","marker":"28"},{"why":"Case study grounding the claim that ML projects fail without software engineering discipline.","marker":"29"},{"why":"DOME recommendations provide the existing validation standards this paper adapts for protein engineering.","marker":"36"},{"why":"Identifies random splits and test-set leakage as a common pitfall the guidelines are designed to prevent.","marker":"38"},{"why":"Defines evolutionary-aware and leakage-avoiding data splitting, a load-bearing component of the workflow.","marker":"51"},{"why":"Sets the context of ML-guided directed evolution that the paper's supervised learning focus builds on.","marker":"19"},{"why":"Makes the reproducibility argument for ML in life sciences, supporting the call for community standards.","marker":"35"},{"why":"REFORMS consensus recommendations underpin the paper's journal-enforcement angle.","marker":"37"},{"why":"Large-scale study on research code quality that supports the link between code practice and reproducibility.","marker":"32"}],"fun_headline_variants":["Protein ML fails without software engineering rigor","Workflow, not algorithms, decides protein ML success","Treat protein ML as software engineering, not model choice","Reproducible protein ML needs disciplined code and data","Protein engineering ML: it's the workflow, not the model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that weak software engineering practice—more than data scarcity, experimental noise, or inherently hard prediction problems—is the main cause of unreliable machine learning in protein engineering; if the dominant failures are elsewhere, the checklist would miss the core issue.","fun_headline_variants_meta":{"raw":{"variants":["Protein ML fails without software engineering rigor","Workflow, not algorithms, decides protein ML success","Treat protein ML as software engineering, not model choice","Reproducible protein ML needs disciplined code and data","Protein engineering ML: it's the workflow, not the model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000137,"raw_usage":{"total_tokens":1098,"prompt_tokens":842,"completion_tokens":256,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":180}},"tokens_in":458,"tokens_out":256,"duration_ms":3152,"temperature":1.0,"reasoning_tokens":180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:37:26.515764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare two pipelines on the same protein engineering dataset with identical experimental labels: one following the paper's full checklist and one using random splits and unversioned code. If the poorly engineered model matches the well-engineered one on evolutionarily held-out test sets across several datasets, the claim that process determines reliability is weakened. Alternatively, a systematic survey showing that most reported failures trace to label noise or insufficient data, not to split leakage or code errors, would falsify the paper's emphasis.","supporting_citations":[],"review_version":1}