{"id":"b6761cb6-a30b-4670-9c78-d364c47e8701","arxiv_id":"2412.01108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"By adding a surface point-cloud encoder to ESM-2 and a GVP structure encoder, S3F and its EVE-augmented version S3F-MSA report state-of-the-art zero-shot protein fitness scores on ProteinGym.","lead":"This paper introduces S3F, a model that predicts how mutations affect protein function by combining sequence, 3D structure, and surface-shape information. On the ProteinGym benchmark it reports the best zero-shot fitness prediction scores to date, with the largest gains on stability and binding assays.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"S3F-MSA's claimed 8.5% improvement over SaProt comes from ensembling with EVE, not from multi-scale surface features; no equivalent ensemble baselines are reported, so the state-of-the-art claim conflates ensemble gains with architectural gains.","rationale":"The reader's weakest assumption focused on the pLDDT threshold (Section 3.5) and whether it was test-set tuned. That is a valid concern, but I find a more directly load-bearing issue: the headline SOTA result is obtained by ensembling S3F with EVE, while the comparison baselines, especially SaProt, are single models. The paper does not provide an ensemble baseline, so the 8.5% improvement cannot be attributed to the proposed multi-scale representation. This does not invalidate the paper entirely—S3F alone (0.470) does beat SaProt (0.457) in the no-MSA category, and the ablation shows surface helps over S2F—but it changes the interpretation of the central claim. The reader's conditional verdict remains appropriate; the paper should be accepted only if the authors provide these ensemble baselines or temper the SOTA claim. I did not find evidence of fit-to-test in the masked pretraining, and the architecture is well described, but the evaluation protocol for the headline comparison is incomplete. The concrete test above would settle the concern directly.","tokens_in":15717,"tokens_out":3945,"duration_ms":35523,"concrete_test":"Re-run the ProteinGym evaluation adding two ensemble baselines using exactly the same z-score summation procedure as S3F-MSA: (1) ESM2-650M + EVE and (2) SaProt + EVE. If either reaches or exceeds Spearman 0.496, then the SOTA claim is not supported by the multi-scale architecture. If S3F-MSA still outperforms both by a margin larger than the bootstrap standard error, the concern is resolved. Also compare S2F-MSA vs S3F-MSA to verify that the surface module adds significant signal after ensembling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract, Section 4.2) is that S3F-MSA achieves state-of-the-art Spearman 0.496, improving on SaProt (0.457) by 8.5%. However, S3F-MSA is defined in Section 4.1 as the z-score sum of S3F and EVE, a strong alignment-based model. SaProt is a single model without an equivalent ensemble baseline. The marginal gains attributable to the proposed multi-scale architecture are small: S2F→S3F adds 0.016 Spearman (0.454→0.470 in the no-MSA category), and S2F-MSA→S3F-MSA adds only 0.009 (0.487→0.496). Without reporting ESM2+EVE or SaProt+EVE ensembles, the unique contribution of sequence-structure-surface representations to the headline SOTA cannot be isolated. The paper's ablation (Table B.3) even shows that removing sequence and structure while keeping surface alone underperforms ESM2 (0.392 vs 0.414), indicating that the surface module is not independently predictive. The abstract's 'state-of-the-art fitness prediction' thus overstates what the proposed architecture alone demonstrates; the 8.5% gain may be largely an artifact of ensembling with EVE, a pre-existing baseline that already achieves 0.439 by itself. This is a load-bearing gap because the paper's core novelty claim is about multi-scale surface features, not about ensembling.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces S2F and S3F, multi-scale protein encoders that combine ESM-2 language-model embeddings with GVP message passing over C-alpha backbone graphs and, for S3F, dMaSIF surface point clouds. The models are pre-trained on CATH with a masked-residue prediction objective and evaluated zero-shot on the ProteinGym benchmark's 217 substitution assays. S2F reaches 0.454 average Spearman correlation, S3F reaches 0.470, and ensembling with EVE yields S2F-MSA at 0.487 and S3F-MSA at 0.496, which the paper claims is state of the art and an 8.5% improvement over SaProt's 0.457. The paper also presents breakdown analyses by assay type, MSA depth, taxon, mutation depth, and structure quality, plus an out-of-distribution generalization experiment.","tokens_in":16038,"tokens_out":6523,"duration_ms":58193,"significance":"If verified, the central contribution is meaningful: S3F improves on ESM-2 by 0.056 Spearman with only 20M trainable parameters, and the OOD generalization check supports the transferability of the learned structure and surface representations. Using the established ProteinGym benchmark and bootstrap-based comparisons is a strength, and the multi-scale framework is modular enough to be adapted to future protein language models. However, the headline 8.5% gain is obtained by ensembling with EVE, and no equivalent ensemble baselines are reported for the comparison models; the pLDDT fallback rule in Section 3.5 is not justified as a fixed a priori design choice; and the code, data, and pretrained weights are not available at submission. The evidence for the specific contribution of surface topology is also modest, as the surface-only ablation in Table B.3 underperforms ESM-2.","major_comments":[{"comment":"The headline comparison \"S3F-MSA improves SaProt by 8.5%\" is not an apples-to-apples test of the proposed architecture. S3F-MSA is defined as the z-score sum of S3F and EVE, while SaProt is a single model with no equivalent ensemble. No ESM-2+EVE, S2F+EVE, or SaProt+EVE baseline is reported, so the 8.5% gain conflates ensemble gains with gains from the multi-scale architecture. Please add the missing ensemble baselines and reframe the state-of-the-art claim to compare within categories, e.g., S3F versus SaProt in the no-MSA category and S3F-MSA versus TranceptEVE in the MSA category.","section":"Section 4.1 and 4.2 (Table 1)"},{"comment":"The pLDDT-based fallback rule (\"for mutations on residues with pLDDT score below 70, we use the output scores from ESM-2-650M\") is load-bearing for the paper's zero-shot claim, but the threshold is not derived or ablated. If the threshold was chosen after inspecting ProteinGym results, the comparison is not strictly zero-shot. State how the threshold was fixed, report sensitivity to the threshold (e.g., 50/60/70/80) on a held-out validation set, and clarify whether the rule is applied per mutated residue or per assay.","section":"Section 3.5"},{"comment":"The reported standard error of 0.000 for SaProt's difference to the best score is implausible and likely a typo. More importantly, Table 2 only reports each baseline's difference to S3F-MSA, not pairwise bootstrap confidence intervals between the models being compared. The sentence \"Our performance delta with prior methods are all statistically significant\" therefore needs pairwise tests, or at least per-model bootstrap errors, to be supported.","section":"Table 2 and Section B.2"},{"comment":"The paper states that code, data, pretrained weights, and statistical significance results \"will be released upon acceptance.\" Because the central empirical claims depend on trained GVP weights and on the exact AlphaFold2-structure handling, the current submission is not reproducible as written. Please provide the code and weights, or at minimum per-assay prediction tables and a complete specification of hyperparameters, surface point sampling, and pLDDT handling.","section":"Checklist items 5 and 7"}],"minor_comments":[{"comment":"The text says 23 out-of-distribution assays are selected, but Figure 4 lists 19 proteins; clarify whether multiple assays share the same protein and how grouping was performed.","section":"Section 4.5 and Figure 4"},{"comment":"The surface point count is stated as 6K-20K points; specify how this depends on protein size and whether it is fixed during inference.","section":"Section 3.4"},{"comment":"The x-axis labels 0 through 4 are described as groups of AlphaFold2-predicted structures ranked by pLDDT, but the ranking procedure is not defined; please state how the five structures are ordered and how ties or missing pLDDT values are handled.","section":"Figure 2(f)"},{"comment":"The ablation shows that removing structure while keeping surface alone drops performance below ESM-2 (0.392 vs 0.414); this negative result is relevant to the claim about surface features and should be discussed in the main text, not only in the appendix.","section":"Section B.3"},{"comment":"Calling SaProt \"the current state-of-the-art method\" in the context of the MSA-augmented S3F-MSA comparison is confusing, because SaProt appears in the no-MSA category of Table 1; reword to avoid mixing categories.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a credible benchmark result and a lightweight, modular architecture, but the two main technical concerns are the uncontrolled ensemble comparison and the unjustified pLDDT fallback rule. Both are fixable with additional experiments and clarifications, so I support major revision rather than rejection. The checklist inconsistency between Section B.2 and item 7 should also be corrected during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this paper is worth reading but the marketing overreaches. The actual result is that adding a surface encoder to a sequence+structure model (S2F->S3F) buys about 0.016 Spearman on ProteinGym zero-shot, and the jump from 0.454 to 0.496 for the MSA variants comes mostly from ensembling with EVE, not from the surface representation. The stress-test note is right: S3F-MSA is S3F plus EVE, and SaProt is reported without an equivalent ensemble. Until we see SaProt+EVE or ESM2+EVE, you can't attribute the 8.5% gain to the proposed architecture.\n\nWhat's genuinely new: a modular, lightweight (20M trainable params) framework that fuses ESM-2 embeddings with GVP on backbone and dMaSIF surface point clouds, pre-trained on CATH with a masked residue prediction loss. The surface point masking during pre-training is a nice touch to avoid leakage. The evaluation is thorough: 217 assays, bootstrapped significance (modulo the implausible 0.000 std error for SaProt in Table 2), breakdown by function, MSA depth, taxon, mutation depth, and structure quality. The GB1 case study on epistasis is illustrative. They also show out-of-distribution generalization to 23 low-similarity assays.\n\nSoft spots, in order of severity:\n\n1. The state-of-the-art claim is inflated by ensembling. The ablation in Table B.3 shows surface features alone (w/o structure) underperform ESM2 (0.392 vs 0.414), so the surface module is not independently predictive. The paper should report ensemble baselines for ESM2 and SaProt, and temper the abstract.\n\n2. The pLDDT threshold of 70 for falling back to ESM-2 is introduced without any a priori justification. If that threshold was chosen after seeing ProteinGym results, the zero-shot comparison is compromised. The authors need to clarify.\n\n3. Missing comparison against Lee et al. [44], which also pretrains sequence-structure-surface representations. The paper cites it but doesn't benchmark against it. This weakens the novelty claim.\n\n4. No code or weights at submission. The checklist says they'll be released on acceptance, but for a benchmark paper, that's a real reproducibility gap.\n\n5. The fixed-backbone and no-side-chain assumptions are stated as limitations; they're reasonable simplifications, but they do cap the potential gains.\n\nOverall: the central contribution—surface topology adds small but real signal for fitness prediction—survives, but the magnitude is modest and the headline overstates it. This is a fair candidate for peer review with major revision: add the missing baselines, justify the threshold, compare to [44], and release artifacts. I'd bring it to a reading group for a discussion of evaluation practices in this area.","headline":"Solid benchmark work showing surface features add a modest gain, but the headline state-of-the-art claim rests on an EVE ensemble, not on the multi-scale architecture alone.","tokens_in":16598,"tokens_out":3385,"would_cite":true,"duration_ms":26481,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that integrating sequence, structure, and surface representations yields the best zero-shot protein fitness predictions on ProteinGym.","keywords":["protein fitness prediction","zero-shot mutation effect prediction","multi-scale representation learning","protein surface","geometric vector perceptron","ProteinGym","deep mutational scanning","CATH"],"falsifier":"Recompute the 217-assay average with the low-confidence fallback threshold removed or varied across a grid; if no pre-specified threshold reproduces a Spearman of 0.496 for S3F-MSA, or if the ranking against SaProt changes with the threshold, the central claim collapses.","tokens_in":15530,"feed_emoji":"🧬","tokens_out":6697,"duration_ms":55507,"temperature":0.7,"pith_summary":"The paper claims that protein fitness prediction improves when a model is trained across three scales at once: amino-acid sequence, backbone structure, and molecular surface. On 217 deep mutational scanning assays from ProteinGym, the full model (S3F-MSA) reaches a Spearman correlation of 0.496, beating the previous best sequence-structure model by 8.5 percent; even without alignment input, the surface-augmented model (S3F) scores 0.470. A sympathetic reader would care because mutation-effect prediction is a bottleneck in protein engineering and clinical variant interpretation, and the paper argues that surface geometry carries signal that sequence and backbone models miss. The method keeps the protein language model frozen and trains only a small structure-surface encoder on 31,000 CATH structures, so the claimed gains come with far fewer trainable parameters than leading baselines.","feed_headline":"Protein surface features push fitness prediction to 0.496 Spearman","feed_subtitle":"S3F-MSA beats the previous best model by 8.5% across 217 ProteinGym deep mutational scanning assays.","key_machinery":"The engine is a two-graph geometric message-passing decoder built on top of frozen ESM-2-650M embeddings. A Geometric Vector Perceptron (GVP), a message-passing layer that operates on rotation-equivariant scalar and vector features, updates residue features over a 10-angstrom radius graph on alpha carbons, and a second GVP updates surface-point features over a 16-neighbor point cloud generated by dMaSIF. Each surface point carries Gaussian curvature and Heat Kernel Signature features, and residue and surface representations are pooled back and forth using three nearest residues per surface point and twenty nearest surface points per residue. The whole stack is pre-trained with BERT-style masked residue prediction on CATH, mutation scores are computed as the log-odds ratio between mutant and wild-type residues at masked positions, and a hybrid inference rule uses ESM-2's score whenever AlphaFold2 confidence (pLDDT) is below 70 and the multi-scale score otherwise.","core_discovery":"The central discovery claimed is that multi-scale representations, namely sequence tokens, wild-type backbone coordinates, and point-cloud surface geometry, combine additively for zero-shot fitness prediction. S2F, the sequence-plus-structure model, roughly matches the best structure-aware baseline, while S3F, which adds the surface encoder, is the best model that does not use MSA input. With alignment-based EVE scores ensembled, S3F-MSA outperforms all 17 baselines on ProteinGym, and the ablation shows that removing the surface encoder drops Spearman from 0.470 to 0.454, while removing the structure encoder drops it to 0.392.","pith_inferences":["Editorial inference: an implication the authors leave implicit is that the surface encoder's benefit should transfer to other zero-shot tasks beyond substitution fitness, such as binding-site or interface prediction, since surface geometry is the same modality.","Editorial inference: a testable extension is to replace the frozen ESM-2 with a newer language model and measure whether the multi-scale gains persist; the paper's modular design predicts they would.","Editorial inference: the low-confidence fallback rule (pLDDT below 70) could be stress-tested by sweeping the threshold; if performance is sensitive to it, the reported zero-shot comparison is less robust than it appears.","Editorial inference: another extension is to adapt the method to insertions and deletions by allowing the backbone graph to change at the mutated sites, which the paper explicitly leaves to future work."],"forward_implications":["Surface augmentation yields the largest gains on binding and stability assays, where structure-related mutations matter most.","The model's advantage grows with mutation depth, which the authors interpret as better capture of epistatic effects.","Because ESM-2 is frozen and only the GVP layers are trained (20M parameters for S3F), the same scaffolding can be re-attached to any future protein language model without retraining the language model.","The 8.5% improvement over SaProt in Spearman is statistically significant under ProteinGym's bootstrap difference test, with a standard error of 0.005 for S3F and 0.000 for S3F-MSA."],"supporting_citations":[{"why":"Supplies the frozen ESM-2-650M sequence embeddings that initialize every node feature in the structure and surface graphs.","marker":"[32]"},{"why":"Establishes the zero-shot mutation scoring rule as the log-odds ratio between mutant and wild-type residue probabilities at masked positions.","marker":"[5]"},{"why":"Provides the Geometric Vector Perceptron message-passing layer used for both the backbone graph and the surface graph.","marker":"[19]"},{"why":"Generates the surface point clouds, with Gaussian curvature and heat kernel signatures, from backbone structures.","marker":"[40]"},{"why":"Supplies the CATH structures and masked-residue pre-training objective on which S2F and S3F are trained.","marker":"[30]"},{"why":"Defines the 217 substitution assays, the Spearman metric, and the bootstrap significance test behind the headline comparison.","marker":"[31]"},{"why":"SaProt is the prior best model the paper compares against, and the 8.5% relative improvement over it is the main claim.","marker":"[26]"},{"why":"EVE's family-specific predictions are z-score summed with S2F and S3F to form the MSA-augmented variants S2F-MSA and S3F-MSA.","marker":"[7]"}],"fun_headline_variants":["Surface-aware protein model achieves state-of-the-art fitness prediction","ProteinGym top score via sequence-structure-surface fusion","S3F integrates surface geometry to beat all baselines on ProteinGym","Protein surface encoder lifts fitness prediction to beat all baselines","Multi-scale protein representation including surface wins ProteinGym"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the rule to fall back to the sequence-only model for low-confidence AlphaFold2 regions was fixed before the authors looked at ProteinGym results, so the reported zero-shot comparison is a fair one.","fun_headline_variants_meta":{"raw":{"variants":["Surface-aware protein model achieves state-of-the-art fitness prediction","ProteinGym top score via sequence-structure-surface fusion","S3F integrates surface geometry to beat all baselines on ProteinGym","Protein surface encoder lifts fitness prediction to beat all baselines","Multi-scale protein representation including surface wins ProteinGym"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000939,"raw_usage":{"total_tokens":3994,"prompt_tokens":904,"completion_tokens":3090,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":3006}},"tokens_in":520,"tokens_out":3090,"duration_ms":21123,"temperature":1.0,"reasoning_tokens":3006,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:41:12.205082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the 217-assay average with the low-confidence fallback threshold removed or varied across a grid; if no pre-specified threshold reproduces a Spearman of 0.496 for S3F-MSA, or if the ranking against SaProt changes with the threshold, the central claim collapses.","supporting_citations":[{"cited_title":"Language models enable zero-shot prediction of the effects of mutations on protein function","cited_arxiv_id":null,"evidence_quote":"Establishes the zero-shot mutation scoring rule as the log-odds ratio between mutant and wild-type residue probabilities at masked positions."},{"cited_title":"Soni, and Ron O","cited_arxiv_id":null,"evidence_quote":"Provides the Geometric Vector Perceptron message-passing layer used for both the backbone graph and the surface graph."},{"cited_title":"Fast end-to-end learning on protein surfaces","cited_arxiv_id":null,"evidence_quote":"Generates the surface point clouds, with Gaussian curvature and heat kernel signatures, from backbone structures."},{"cited_title":"Cath: an expanded resource to predict protein function through structure and sequence","cited_arxiv_id":null,"evidence_quote":"Supplies the CATH structures and masked-residue pre-training objective on which S2F and S3F are trained."},{"cited_title":"Proteingym: large- scale benchmarks for protein fitness prediction and design","cited_arxiv_id":null,"evidence_quote":"Defines the 217 substitution assays, the Spearman metric, and the bootstrap significance test behind the headline comparison."},{"cited_title":"Saprot: Protein language modeling with structure-aware vocabulary","cited_arxiv_id":null,"evidence_quote":"SaProt is the prior best model the paper compares against, and the 8.5% relative improvement over it is the main claim."}],"review_version":1}