{"id":"030fb4ed-2d2b-4fec-b2d2-1c78fc5ffc34","arxiv_id":"2412.04526","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Using the ESM3 protein language model with sequence and structure inputs, the authors report a Pearson correlation of 0.50 on the s571 DeltaTm benchmark, a slight improvement over prior SOTA of 0.47.","lead":"The paper trains a model that predicts how much a single mutation changes a protein's melting temperature, using both the protein's sequence and its predicted 3D structure. It reports the best accuracy yet on a standard set of 571 test mutations, though the improvement is small.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ESM3-DTm's SOTA claim may be confounded: it is fed ColabFold mutant structures while GeoDTm-3D is not, so the 0.03 PCC gain may reflect an extra input channel.","rationale":"The reader's verdict was CONDITIONAL, and I agree with that verdict. I did not find a fatal flaw; the paper is a plausible empirical contribution. But the weakest link in the central SOTA claim is the uncontrolled input channel: ESM3-DTm receives a predicted structure of the mutant, while the quoted GeoDTm-3D baseline does not. The paper explicitly describes generating PDB structures for all mutated structures with ColabFold (Data section) and ESM3-DTm as accepting both sequence and structure (Model Setup). The baseline results are quoted from Xu et al. 2023 and no indication is given that GeoDTm-3D was offered mutant structures. Therefore the 0.50 vs 0.47 PCC could reflect extra information rather than a better model. This is the single most load-bearing concern because the entire paper's headline is the SOTA claim. The proposed ablation (wild-type-only ESM3-DTm, or mutant-structure-augmented GeoDTm-3D) would settle it. Secondary concerns—missing confidence intervals and lack of a sequence-identity check between s571 and s4346—reinforce the need for caution; the MMseqs2 split is only used for training/validation, not to de-duplicate the test set. None of these are grounds for rejection, but they justify the CONDITIONAL verdict already given. I therefore recommend keeping the reader's verdict unchanged, with the added explicit condition that the authors run the controlled comparison above.","tokens_in":7550,"tokens_out":5928,"duration_ms":54443,"concrete_test":"Run a controlled ablation on the s571 test set: (1) retrain/evaluate ESM3-DTm with only wild-type structures (no mutant PDB, using the same sequence and wild-type structure inputs); or (2) provide GeoDTm-3D with the same ColabFold-predicted mutant PDBs as additional/alternative structural input. If ESM3-DTm's PCC drops to ≤0.47 in (1), or GeoDTm-3D's PCC rises to ≥0.50 in (2), the reported SOTA improvement is attributable to the mutant-structure input asymmetry rather than to the multi-modal representation. Report bootstrapped 95% CIs for all PCC values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ESM3-DTm (PCC 0.50) beats GeoDTm-3D (PCC 0.47) on s571. That comparison is only meaningful if both models receive the same information. In the Data section the authors state that 'for all mutated structures, we generated PDB structures using ColabFold,' and ESM3-DTm is described as accepting 'both sequence and PDB structure inputs' (Figure 1b, Algorithm 1). GeoDTm-3D is a geometric method that, per the cited work (Xu et al. 2023), uses wild-type structures; no mutant structure is supplied to it. Thus ESM3-DTm has access to a ColabFold-predicted structure of the very mutant whose ΔTm is being predicted, an input that encodes the mutation and may contain stability-relevant structural changes. This is an information asymmetry, not a modeling improvement. The advantage is large enough to matter: a PCC gap of 0.03 could easily be explained by the extra mutant-structure channel, especially with no confidence intervals reported. If GeoDTm-3D were given the same mutant structures, or if ESM3-DTm were restricted to wild-type structure, the SOTA margin might vanish. The paper's 'fair evaluation' claim in the abstract therefore needs an explicit baseline-controlled test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ESM3-DTm, a regression model for predicting changes in protein melting temperature (ΔTm) upon single-point mutations. The model fine-tunes the ESM3-1.4B multimodal language model on both wild-type and mutated protein sequences and, in its full form, on predicted PDB structures of the mutant, using a regression head that combines an outer-product of mutated-position embeddings with a linear combination of CLS embeddings. Training uses the s4346 set and evaluation uses the s571 test set from the GeoStab benchmark. The authors report a Pearson correlation of 0.50, MAE of 5.21, and RMSE of 7.68, which they claim is state-of-the-art compared with GeoDTm-3D (PCC 0.47). The paper also compares ESM2, SaProt, and OpenFold backbones under nominally similar settings and includes ablations of regression heads and fine-tuning strategies.","tokens_in":7851,"tokens_out":5997,"duration_ms":57607,"significance":"If the reported comparison is sound, the paper would provide a useful demonstration that multimodal protein language model representations can improve ΔTm prediction, a task with relatively few deep-learning baselines. The use of an external test set (s571), the systematic comparison of several backbone models, and the ablation studies are strengths that make the central claim easy to scrutinize. However, the headline SOTA claim is currently supported only by a small margin over a baseline that does not receive the same mutant-structure information, with no uncertainty quantification. The paper is therefore of moderate significance; the result is plausible but not yet established.","major_comments":[{"comment":"The SOTA comparison is confounded by an information asymmetry. The Data section states that 'for all mutated structures, we generated PDB structures using ColabFold,' and ESM3-DTm is described as accepting both sequence and PDB structure inputs. GeoDTm-3D, the baseline to which the headline PCC improvement is compared, is a structure-based method that, per the cited work (Xu et al. 2023), uses wild-type structures and is not given the mutant structure. Thus ESM3-DTm has access to a predicted structure of the very mutant whose ΔTm is being predicted, which is an additional input channel that may encode stability-relevant information independent of the model architecture. The reported margin (PCC 0.50 vs. 0.47) could plausibly be explained by this extra channel. Please provide a controlled comparison: either supply the same mutant structures to GeoDTm-3D (or an equivalent structure-based baseline), or restrict ESM3-DTm to wild-type structures only, and report whether the PCC gap persists.","section":"Data; Algorithm 1; Table 1"},{"comment":"No uncertainty quantification is provided for the headline metrics. The improvements over GeoDTm-3D are small in absolute terms (0.03 PCC, 0.10 MAE, 0.35 RMSE), and the 571 test mutations are distributed across only 37 proteins, so the effective sample size is considerably smaller than 571. Without confidence intervals, bootstrap estimates, or a paired significance test, the claim that ESM3-DTm is statistically superior to GeoDTm-3D is not supported. Please report per-protein or bootstrapped intervals and a paired comparison of the PCC difference.","section":"Results; Tables 1 and 2"},{"comment":"The 'fair evaluation' between backbones is not actually controlled. OpenFold-DTm freezes the backbone and trains only the linear layer, while ESM2, ESM3, and SaProt backbones are fully fine-tuned; ESM3-DTm additionally receives mutant structures, while ESM2-DTm and ESM3-DTm (seq only) do not. The performance differences in Table 2 therefore conflate architecture, input modality, and fine-tuning strategy. Please equalize the training regimes (e.g., fine-tune all backbones or freeze all backbones) and control the structure input before drawing conclusions about the relative merits of the backbone models.","section":"Experiments: Model Setup; Table 2"},{"comment":"The outer-product regression head is written with a linear layer of O(d^3) parameters. In Algorithm 1, Head1 = Flatten(am ⊗ aw ∈ R^{d^2}) with a learned map W: R^{d^2} → R^d gives a weight matrix of dimension d^2 × d = d^3. For a multimodal hidden dimension on the order of 1536, this layer would contain roughly 3.6 billion parameters, exceeding the size of the ESM3-1.4B backbone. Please clarify whether this is the actual implementation, and if so, describe the regularization used to control overfitting; alternatively, correct the notation if a lower-dimensional projection or low-rank approximation is employed.","section":"Algorithm 1"},{"comment":"There is no sequence-identity control between the training and test sets. The authors use MMseqs2 at 50% identity to create the validation split, but they do not report whether proteins in s571 share high sequence identity with proteins in s4346. If such overlap exists, the reported metrics would be optimistic relative to a truly held-out setting. Please report the maximum pairwise sequence identity between each s571 protein and the s4346 training set, and consider presenting results on a subset clustered at, for example, 30% identity.","section":"Data; data split"}],"minor_comments":[{"comment":"The abstract mentions AlphaFold, but the experiments use OpenFold as the structure-prediction backbone; please align the terminology.","section":"Abstract"},{"comment":"The definitions of N1 and N2 are incomplete: the algorithm states that N1 and N2 are linear layers connected after Head1 and Head2, but their output dimensions are not specified.","section":"Algorithm 1"},{"comment":"Table 4 lacks the upward/downward arrows used in other tables to indicate whether higher or lower values are better; please make the table headers consistent.","section":"Table 4"},{"comment":"The cross-reference 'in Section' in the Model Setup paragraph is unresolved; please insert the correct section number.","section":"Regression Head"},{"comment":"No code or data availability statement is provided; given the central role of ColabFold-generated structures and ESM3 inference, a reproducibility statement would substantially strengthen the paper.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an applied ML venue, and the central idea of using a multimodal PLM for ΔTm prediction is worth pursuing. The main risk is that the SOTA claim rests on an uncontrolled comparison, so I would ask the authors to add the controlled baseline and uncertainty estimates before any acceptance decision. I would not reject on the current evidence, but the manuscript is not yet publishable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper applies ESM3 to ΔTm prediction and reports a new SOTA (PCC 0.50 vs 0.47 for GeoDTm-3D). The stress-test note is right: that comparison is not apples-to-apples. ESM3-DTm takes both sequence and PDB structure inputs, and the authors generated ColabFold structures for all mutated proteins. GeoDTm-3D, per the cited work, uses only wild-type structures. So ESM3-DTm gets a predicted structure of the very mutant whose ΔTm it is predicting, an input that already encodes the mutation and any stability-relevant structural shift. A 0.03 PCC gain could easily come from that extra channel, and with no confidence intervals we cannot tell. The abstract's phrase \"fair evaluation\" does not survive contact with this setup.\n\nThat said, the paper does real work. The systematic comparison of ESM2, ESM3, SaProt, and OpenFold backbones under similar training protocols is useful, and the authors were careful to build a training/validation split with MMseqs2 at 50% sequence identity, which is more than some prior work bothered with. The ablations of regression heads and fine-tuning vs frozen backbones are informative. The external s571 test set is independent, and the reported metrics support the ranking among the models they trained themselves.\n\nThe soft spots beyond the mutant-structure asymmetry are several but manageable. No error bars or significance tests anywhere. The OpenFold comparison freezes the backbone while the PLMs are fully fine-tuned, which is not a fair representation comparison. Algorithm 1 is cryptic: the outer-product head is O(d^3) in parameters, which for ESM3's hidden dimension is enormous, and the paper never discusses whether that head is the reason ESM3 wins or just a memory hazard. No code or data released, so none of this is independently reproducible. They also never check sequence identity between s571 and their training set, so leakage cannot be ruled out, though the external origin makes it less likely.\n\nMy bottom line: the paper is an honest empirical exploration, but the headline SOTA claim is not established. The right fix is an ablation where ESM3-DTm is given only wild-type structure (or none), and ideally confidence intervals from multiple seeds. If the margin over GeoDTm-3D vanishes, the paper is still a reasonable benchmark comparison, just not a SOTA claim.\n\nI would not desk-reject it. The task is niche but practical, the experiments are mostly sensible, and the issues are addressable in revision. I would send it to review with a specific request for the wild-type-only control and error bars.","headline":"The SOTA claim is confounded: ESM3-DTm sees ColabFold-predicted mutant structures while GeoDTm-3D does not, so the 0.03 PCC gap may be an input channel, not a better model.","tokens_in":8326,"tokens_out":1672,"would_cite":false,"duration_ms":18548,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ESM3-DTm, a model built by fine-tuning the multimodal protein language model ESM3 on sequence and PDB-style structure inputs, predicts the change in protein melting temperature caused by a single-point mutation…","keywords":["protein melting temperature","Delta Tm prediction","protein language model","ESM3","multimodal protein representation","protein stability","mutation effect prediction","fine-tuning"],"falsifier":"Re-run the comparison with identical wild-type and mutant structures provided to ESM3-DTm and GeoDTm-3D, and compute bootstrap or repeated-seed confidence intervals on the Pearson correlation for the 571 test mutations; if those intervals overlap with 0.47, or if a sequence-identity check shows substantial overlap between the s4346 training set and s571 test set, the claimed improvement is not established.","tokens_in":7375,"feed_emoji":"🧬","tokens_out":9400,"duration_ms":82997,"temperature":0.7,"pith_summary":"This paper claims that a protein language model that reads both amino-acid sequence and structure, fine-tuned end-to-end on a curated set of single-point mutations, predicts $\\Delta T_m$ (the change in melting temperature caused by a mutation) better than any previously published method on the shared s571 benchmark. Their model, ESM3-DTm, built on the 1.4-billion-parameter ESM3, reports a Pearson correlation of 0.50, a mean absolute error of 5.21, and a root mean square error of 7.68 on 571 test mutations. The authors also run a fair comparison across ESM2, ESM3, SaProt, and OpenFold backbones to isolate which representation choices matter. The result matters because melting temperature is a direct experimental readout of protein stability, so better prediction would let protein engineers rank candidate mutations before going to the lab.","feed_headline":"Multimodal model tops protein melt-temperature scoreboard at 0.50","feed_subtitle":"Sequence-plus-structure embeddings edge out earlier predictors and point to where stability modeling can improve.","key_machinery":"The load-bearing object is ESM3-DTm, built on ESM3-1.4B, a multimodal protein language model that encodes each protein both as an amino-acid sequence and as a PDB-style structure. For a mutation, the model extracts a CLS summary embedding and a mutated-position token embedding for both the wild-type and mutant protein, then feeds them into two regression heads: one computes the outer product of the mutated-position embeddings, and the other concatenates layer-normalized differences of the CLS and position embeddings. The two heads' predictions are averaged, and the whole network is fine-tuned end-to-end with a mean-squared-error loss. The comparison variants (ESM2-DTm, SaProt-DTm, OpenFold-DTm) share the same two-head regression framework with their respective feature extractions.","core_discovery":"The paper's central claim is that jointly using sequence and structure through a multimodal, fully fine-tuned protein language model yields state-of-the-art $\\Delta T_m$ prediction. On the s571 test set, ESM3-DTm surpasses the best prior structure-based method GeoDTm-3D (Pearson correlation 0.50 versus 0.47) and the best sequence-based method GeoDTm-Seq (0.46), while also beating its own sequence-only ESM3 variant (0.49) and an ESM2-based version (0.48). The authors attribute the gain to the multimodal ESM3 backbone and to fine-tuning the entire model rather than freezing the backbone, and they show that the choice of regression head substantially changes accuracy.","pith_inferences":["If the small sequence-versus-multimodal gap (0.49 versus 0.50) survives fresh evaluation, the practical lesson is that most of the predictive signal lives in the language model and the regression setup, with structure adding only a modest increment on this benchmark.","The same two-head ensemble could be transferred to other mutation-effect regression tasks such as $\\Delta\\Delta G$ or enzyme kinetic parameters, where multimodal ESM3 embeddings may behave differently than on melting temperature.","A decisive robustness test would be to split the available mutations by sequence clusters at 30% identity and retrain from scratch, reporting confidence intervals; that would show whether the reported ordering of backbones is stable or tied to the particular s4346/s571 partition."],"forward_implications":["On the shared s571 benchmark, ESM3-DTm reports the best published $\\Delta T_m$ prediction: Pearson correlation 0.50, MAE 5.21, and RMSE 7.68.","Multimodal input helps: ESM3-DTm (0.50) outperforms the same model with sequence-only input (0.49) and the ESM2-based model (0.48).","Fully fine-tuning the backbone is better than freezing it: ESM2-DTm improves from Pearson 0.46 to 0.48 when the backbone is trained.","Regression-head design matters: combining mutated-position outer product with CLS and position differences (0.41/0.40) far outperforms simple token concatenation (0.21) on the ESM2 backbone."],"supporting_citations":[{"why":"Supplies the ESM3-1.4B multimodal backbone that accepts both sequence and PDB structure inputs and is fine-tuned into ESM3-DTm.","marker":"Hayes et al. 2024"},{"why":"Defines the s4346 training set and s571 test set and supplies the GeoDTm-3D and GeoDTm-Seq baseline numbers the paper must beat.","marker":"Xu, Liu, and Gong 2023"},{"why":"Provides the ESM2-650M sequence-only backbone used for ESM2-DTm and for the regression-head and fine-tuning ablations.","marker":"Lin et al. 2022"},{"why":"Provides SaProt-650M, the structure-aware sequence vocabulary backbone used as a comparison model.","marker":"Su et al. 2023"},{"why":"Provides OpenFold, the structure-prediction backbone whose frozen features are used for the OpenFold-DTm comparison.","marker":"Ahdritz et al. 2024"},{"why":"Provides ColabFold, which generates the MSAs and the PDB structures for proteins and mutants that lack experimental structures.","marker":"Mirdita et al. 2022"},{"why":"Provides MMseqs2, used to split the training mutations at 50% sequence identity into training and validation sets.","marker":"Steinegger and Söding 2017"},{"why":"Supplies ThermoMutDB, one of the two experimental mutation databases from which the training and test mutations are drawn.","marker":"Xavier et al. 2021"}],"fun_headline_variants":["Multimodal ESM3 predicts protein melt shifts with 0.50 correlation","Protein melt prediction hits 0.50 PCC with multimodal ESM3","Multimodal model beats prior protein stability predictors","Fusing sequence and structure improves melt temperature forecasts","ESM3 fine-tune sets new high for protein melt prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the assumption that the reported 0.50 Pearson correlation is genuinely better than the earlier predictor's 0.47 and is not a fluke of evaluation noise, of predicted mutant structures the earlier method never saw, or of training proteins too similar to test proteins; the paper gives no confidence intervals and no sequence-overlap check.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal ESM3 predicts protein melt shifts with 0.50 correlation","Protein melt prediction hits 0.50 PCC with multimodal ESM3","Multimodal model beats prior protein stability predictors","Fusing sequence and structure improves melt temperature forecasts","ESM3 fine-tune sets new high for protein melt prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3510,"prompt_tokens":833,"completion_tokens":2677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":2590}},"tokens_in":449,"tokens_out":2677,"duration_ms":18285,"temperature":1.0,"reasoning_tokens":2590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:33:23.399053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with identical wild-type and mutant structures provided to ESM3-DTm and GeoDTm-3D, and compute bootstrap or repeated-seed confidence intervals on the Pearson correlation for the 571 test mutations; if those intervals overlap with 0.47, or if a sequence-identity check shows substantial overlap between the s4346 training set and s571 test set, the claimed improvement is not established.","supporting_citations":[{"cited_title":"J.; Oktay, D.; Lin, Z.; Verkuil, R.; Tran, V","cited_arxiv_id":null,"evidence_quote":"Supplies the ESM3-1.4B multimodal backbone that accepts both sequence and PDB structure inputs and is fine-tuned into ESM3-DTm."},{"cited_title":"J.; Berenberg, D.; Fisk, I.; Zanichelli, N.; et al","cited_arxiv_id":null,"evidence_quote":"Provides OpenFold, the structure-prediction backbone whose frozen features are used for the OpenFold-DTm comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ColabFold, which generates the MSAs and the PDB structures for proteins and mutants that lack experimental structures."},{"cited_title":"S.; Nguyen, T.-B.; Karmarkar, M.; Portelli, S.; Rezende, P","cited_arxiv_id":null,"evidence_quote":"Supplies ThermoMutDB, one of the two experimental mutation databases from which the training and test mutations are drawn."}],"review_version":1}