{"id":"92d2c426-1416-4398-9c3f-5633cea05c5e","arxiv_id":"2412.20014","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A CLIP-style protein-text model with new functional segment losses and a large curated dataset reports state-of-the-art scores on 22 protein benchmarks.","lead":"ProtCLIP pre-trains a protein language model on 251.5 million protein-text pairs from UniProt, adding two new 'functional segment' losses on top of the standard CLIP alignment. The resulting model reports state-of-the-art scores on 22 protein benchmarks, but the dataset and code are not released, and the paper does not rule out overlap between pre-training and benchmark test proteins.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ProtAnno-D is built from SwissProt/trEMBL with no stated deduplication against benchmark test sets; if test proteins appear in pre-training, the reported SOTA gains on GO/DeepLoc/FLIP/PPI could reflect memorization rather than transferable function understanding.","rationale":"The reader's conditional verdict already rests on the same load-bearing concern: possible overlap between ProtAnno-D and benchmark test sets. I agree this is the single most important threat to the central claim. All three pillar results - 75% cross-modal improvement, 59.9% GO-CC, and 39.7% GO-BP - are evaluated on UniProt-derived data, and Section 2.1 explicitly describes pre-training data from SwissProt/trEMBL without any deduplication step. Since the text side of ProtAnno-D contains the very annotations used as labels in GO and localization benchmarks, memorization is a concrete failure mode, not a generic worry. The other concerns in the reader's report (no code/weights, missing ProteinCLIP comparison, hyperparameters tuned on validation) are secondary: they affect completeness or reproducibility, but only leakage could invalidate the SOTA numbers themselves. I do not see an internal contradiction that would force a stronger verdict; the ablation tables have minor inconsistencies (e.g., Table 6 'Our sampling strategy' vs Table 7 'Full loss' do not match), but that is not enough to reject the main result. The appropriate disposition remains conditional on release of artifacts and a quantitative leakage analysis.","tokens_in":17146,"tokens_out":8414,"duration_ms":75288,"concrete_test":"Release the accession list for ProtAnno-D and the exact sequence-level test splits for GO/EC, DeepLoc, TAPE, FLIP, SHS27K/148K, and STRING; run MMseqs2 to compute maximum identity between each test protein and ProtAnno-D at 100%, 90%, 50%, and 30% thresholds. Then retrain ProtCLIP after removing every ProtAnno-D entry with >=30% identity to any test protein and rerun all 22 benchmarks; if the reported GO-CC, GO-BP, and cross-modal improvements shrink substantially, the SOTA claim is leakage-driven rather than evidence of transferable function understanding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim requires that pre-training on ProtAnno-D teaches generalizable protein-function alignment, not that the model has seen the answer for each test protein. Section 2.1 says ProtAnno-D is derived from SwissProt and trEMBL and contains 251.5M protein-text pairs, with no identity or homology filtering relative to downstream benchmarks. Sections 3.2-3.6 evaluate on GO/EC, DeepLoc, TAPE, FLIP, and SHS/STRING PPI datasets, whose proteins are also drawn from UniProt-derived resources. The paired text in ProtAnno-D explicitly includes FUNCTION, SUBCELLULAR LOCATION, and SIMILARITY fields (Appendix A), so an exact or near-exact overlap would give the model the benchmark label in the text modality. This is the most direct threat to the two headline numbers: GO-CC (59.9% improvement) and GO-BP (39.7% reported improvement) both use GO annotations that appear in SwissProt/trEMBL entries. Homology at family level can also transfer annotations, so even removing exact duplicates is insufficient. Because no code, data, or weights are released, the manuscript currently contains no evidence that would rule out this memorization route, and the claimed SOTA status is therefore unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ProtCLIP, a multi-modal protein-text pre-training framework that aligns protein sequences with functional textual annotations. It contributes a new large-scale paired dataset ProtAnno (with a dense version containing 251.5M pairs), a property-driven sampling strategy to handle noisy machine-annotated data, and a function-informed pre-training paradigm that combines a global contrastive loss, masked reconstruction of static segments (BSR), property-grouped dynamic segment alignment (PDA), and masked language modeling. The authors report state-of-the-art results on 22 benchmarks across five task types: protein classification engineering, mutation effect prediction, cross-modal transformation, semantic similarity inference, and protein-protein interaction prediction. The paper claims remarkable improvements, including 75% average improvement in cross-modal transformation, 59.9% in GO-CC, and 39.7% in GO-BP protein function prediction.","tokens_in":17467,"tokens_out":8924,"duration_ms":85658,"significance":"If the empirical results are reproducible and not inflated by data leakage, ProtCLIP would be a substantial step forward in protein multi-modal representation learning: the dataset scale (251.5M protein-text pairs) is comparable to image-text pre-training corpora, the sampling strategy addresses a real problem of noisy annotations, and the segment-wise objectives aim at fine-grained functional understanding that prior protein-text models such as ProtST lack. However, the current manuscript does not provide code, data, or model weights, and the absence of deduplication analysis against downstream benchmarks leaves the central SOTA claim unverified. The novelty of the method is interesting and the breadth of evaluation is impressive, but the reliability of the headline numbers is the decisive issue.","major_comments":[{"comment":"The pre-training corpus ProtAnno-D is built from SwissProt and trEMBL, and the paired text explicitly includes 'FUNCTION', 'SUBCELLULAR LOCATION', and 'SIMILARITY' fields. Downstream benchmarks such as GO/EC, DeepLoc, TAPE, FLIP, SHS27K/SHS148K, and STRING are also derived from UniProt or related protein databases. The manuscript does not describe any identity or homology filtering between ProtAnno-D and the benchmark test sets. If the same or homologous proteins appear in both, the reported gains (e.g., GO-CC +59.9% and GO-BP +39.7% in Table 3) could reflect memorization of benchmark labels during pre-training rather than transferable function understanding. Please quantify the overlap (exact and at homology/family level) between ProtAnno-D and each benchmark test set, and re-evaluate after removing overlapping entries; the central SOTA claim depends on this analysis.","section":"Section 2.1, Appendix A, Tables 2-5"},{"comment":"The key pre-training hyperparameters θ=0.3, λ1=0.7, λ2=0.3 are selected via ablations on downstream benchmarks (Sub, EC, Prot2MF, and Sub again) that are also included in the main reported results. This constitutes test-set tuning for those benchmarks and undermines the claim of consistent SOTA across all 22 benchmarks. The authors should select hyperparameters on held-out validation splits (or a separate pre-training validation set), fix them before evaluating on any test set, and clearly document that the reported test numbers were not used for model or hyperparameter selection.","section":"Section 3.1, Section 3.7, Appendix C.4, Figures 6-7"},{"comment":"The PDA objective is underspecified. Equation (6) defines each dynamic segment e_i as a set of thresholded residue vectors, but Equation (7) computes cosine similarity between e_i and the property description embedding a_i, which requires e_i to be a single vector. The paper does not state how e_i is aggregated (e.g., sum, mean, attention pooling) or how gradients are handled through the hard threshold in Equation (5). Please provide the exact vectorization of e_i and the gradient estimation method; without this, the PDA loss and its ablations are not reproducible.","section":"Equations (6)-(7)"},{"comment":"The two summed terms in LPDA are written identically. In a standard symmetric InfoNCE loss, the second term should use the text embedding a_i as the query against all protein segment embeddings e_k, while the first term uses e_i as the query against all text embeddings a_k. As written, the loss is not symmetric and may reduce to a doubled one-directional objective. Please correct the equation and verify that the implemented loss matches the intended bidirectional alignment.","section":"Equation (7)"},{"comment":"The manuscript provides no code, data, or trained model weights, and the ProtAnno dataset is not released. Since the central contributions are a new pre-training dataset and a new model, the empirical claims cannot be independently checked. Please include an availability statement and, at minimum, release the deduplicated pre-training data and model weights (or a representative subset) along with evaluation scripts, so that the reported benchmark results are verifiable.","section":"Abstract, Section 2-6"}],"minor_comments":[{"comment":"The abstract reports a 39.7% improvement in GO-BP, but the numbers in Table 3 imply a 65.8% AUPR improvement and a 19.1% Fmax improvement over ProtST-ESM2; please clarify how the 39.7% figure is computed.","section":"Abstract vs Table 3"},{"comment":"The description of the cross-modal transformation setup is incomplete; please specify how the transformation module is trained, what negative samples are used, and how the baseline KG methods are configured, so that the comparison is fair.","section":"Section 3.4"},{"comment":"Figure 5 is not referenced clearly in the text and the two curves ('No loss weights assigned' and 'Loss weights λ1=0.7, λ2=0.3') are not labeled with a legend; please add a clear caption, axis labels, and legend.","section":"Section 3.7, Figure 5"},{"comment":"The header contains 'AA' and 'V' as separate column names; these should be merged into 'AAV' to avoid confusion with amino acid abbreviations.","section":"Table 2 header"},{"comment":"Figure 7 lacks a legend and the y-axis label is partially cut off; also specify the metric (Sub accuracy) in the caption, as the current text says 'Accuracy' without indicating the benchmark.","section":"Appendix C.4, Figure 7"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper's empirical claims are currently unverifiable because of the missing deduplication analysis and the lack of released code/data/weights. I would strongly encourage you to require the authors to provide exact and homology-level overlap statistics between ProtAnno-D and each benchmark test set, and to re-report results after removing overlapping proteins. If substantial overlap exists, the SOTA claims would need major revision. The test-set tuning of hyperparameters (θ, λ1, λ2) is also a serious methodological issue that should be addressed before publication. The methodological ideas are interesting, but the evidence as presented does not yet support the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhou et al. report ProtCLIP, a protein-text foundation model pre-trained on ProtAnno-D, a new 251.5M-pair dataset, with two segment-wise objectives (BSR and PDA) added to the usual CLIP and MLM losses. The dataset is the most useful thing here: it is an order of magnitude larger than ProtST's curated pairs, and the property-driven sampling strategy is a sensible response to the noise in trEMBL. The paper does broad evaluation across 22 benchmarks and includes ablations showing the new losses matter. The authors are upfront that the dense data is machine-annotated. If the numbers hold, this is a solid incremental advance in protein multi-modal learning.\n\nThe soft spots are significant. The most serious is leakage. ProtAnno-D is built from SwissProt/trEMBL, and the paired text includes FUNCTION, SUBCELLULAR LOCATION, and SIMILARITY. The downstream benchmarks GO, DeepLoc, TAPE, FLIP, and the PPI datasets draw from the same UniProt resources. The paper describes no identity or homology filtering between pre-training and test splits. For GO-CC and GO-BP, where the gains are 59.9% and 39.7%, the model may simply be reading the label from the text modality. That is a load-bearing gap, not a minor omission.\n\nOn top of that, no code, data, or weights are released, so there is no way to check or reproduce the numbers. Several hyperparameters, including the loss weight lambda1 and the PDA threshold theta, are selected on downstream validation, which weakens the benchmark claims. And ProteinCLIP (Wu et al. 2024) is cited in the intro but never compared, even though it is the most directly related baseline. The cross-modal transformation results are against KG embedding baselines and BioBridge, not against other protein-text models, so the '75% improvement' is less impressive than it sounds.\n\nThat said, the central method is plausible. The objectives are grounded in a reasonable protein-function motivation, and the ablations suggest they contribute. The paper is not circular and does not rely on self-citation. I would send it to review, but with a strong request for artifacts and a leakage analysis. As is, the SOTA claim is unverified. The reader's take is about right, though I would put slightly more weight on the missing ProteinCLIP comparison.\n\nWho is this for? People building protein-text models and the UniProt-curation crowd. It deserves a serious referee, but only with the expectation of heavy revision.","headline":"A plausible protein-text pre-training paper with a genuinely large new dataset and two sensible new objectives, but the headline SOTA numbers are unverifiable as written due to missing artifacts and a serious benchmark-leakage risk.","tokens_in":18007,"tokens_out":2551,"would_cite":false,"duration_ms":25089,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProtCLIP claims that a function-informed, segment-aware protein-text pre-training scheme beats all compared baselines on every one of 22 protein benchmarks.","keywords":["protein foundation model","multi-modal pre-training","protein-text alignment","protein function prediction","contrastive learning","Gene Ontology prediction","protein language model","noisy-label sampling"],"falsifier":"Check whether ProtCLIP's advantage survives deduplication: exclude from ProtAnno-D every protein with, say, 30% or higher sequence identity to any test protein in GO, DeepLoc, TAPE, FLIP, SHS27K, SHS148K, and STRING, retrain, and compare benchmark scores; if the reported SOTA margins mostly disappear, the gains come from memorizing near-duplicates rather than from transferable function understanding.","tokens_in":16938,"feed_emoji":"🧬","tokens_out":9897,"duration_ms":91845,"temperature":0.7,"pith_summary":"ProtCLIP aims to make protein foundation models work the way CLIP-style image-text models do, by aligning protein sequences with written functional descriptions. The paper's claim is that previous attempts fell short for two fixable reasons: they underused large, noisy, machine-annotated protein-text data, and they aligned only whole sequences and whole texts, ignoring the local segments that actually determine protein function. ProtCLIP addresses both with a 251.5-million-pair dataset (ProtAnno-D) sampled by a property-driven rule, and with two segment-level objectives that reconstruct masked static segments and align property-grouped dynamic segments to text. On 22 benchmarks spanning function and localization classification, mutation effect prediction, cross-modal transformation, semantic similarity, and protein-protein interaction, the paper reports best-reported results, including 59.9% and 39.7% improvements on Gene Ontology cellular-component and biological-process prediction. If correct, this is a broadly better protein representation for downstream biology, not just a marginal gain on one task.","feed_headline":"ProtCLIP tops 22 protein benchmarks with segment-aware training","feed_subtitle":"Function-informed training on 251M noisy protein-text pairs lifts Gene Ontology and cross-modal retrieval.","key_machinery":"The load-bearing mechanism is a four-term pre-training objective. A global contrastive loss (GC) aligns whole protein sequences with whole biotext descriptions. Biotext-guided Static segment Reconstruction (BSR) masks roughly 15% of the sequence as contiguous static segments and reconstructs them through cross-attention to the biotext. Property-grouped Dynamic segment Alignment (PDA) computes residue-to-property similarity weights, sparsifies them with a threshold $\\theta$, forms property-grouped dynamic segments, and contrasts those segments with property prototypes. A protein masked-language-modeling loss (MLM) preserves unimodal knowledge, with weights $\\lambda_1+\\lambda_2=1$ balancing segment and token reconstruction. On the data side, a property-driven sampling rule selects pre-training pairs with probability proportional to $P \\propto C^{-3}\\sqrt{R}N$, where $C$ is annotation confidence (lower is more reliable), $R$ is property coverage, and $N$ is cluster size; this is what lets ProtCLIP exploit the noisy 251.5-million-pair ProtAnno-D rather than the smaller curated set.","core_discovery":"The central claim, on the paper's own terms, is that a protein multi-modal foundation model can be built by explicitly modeling the two ways functional regions appear in a sequence. Static functional segments are contiguous runs of residues; dynamic segments are residue sets flexibly selected by their similarity to each property description. ProtCLIP learns both: BSR reconstructs masked static segments using biotext as guidance, and PDA groups residues by property prototypes and aligns those groups to the corresponding property descriptions. Combined with global contrastive alignment and masked language modeling, and trained on noisy large-scale data with property-driven sampling, the model outperforms all compared baselines on all 22 benchmarks. The paper interprets this as evidence that fine-grained, function-informed supervision is what was missing from earlier protein-text pre-training.","pith_inferences":["Our inference: the same two-objective design—reconstruct masked contiguous segments and align threshold-selected dynamic segments to text—is portable to other sequence-plus-annotation domains, such as DNA regulatory elements or RNA motifs, where function also depends on localized sequence features.","Our inference: the sampling formula's exponents are not derived from first principles; a systematic sweep over the confidence and coverage exponents would clarify how much of the gain comes from confidence weighting versus coverage weighting, and likely transfer to other noisy multimodal datasets.","Our inference: the headline gains should be read as upper bounds until the pre-training and test corpora are checked for sequence-level overlap; if overlap exists, a deduplicated retraining would give the honest transfer numbers."],"forward_implications":["If ProtCLIP's results hold, noise-heavy machine-annotated protein databases can be converted into effective pre-training signal rather than being discarded for quality.","The segment-wise objectives inject functionally local information that global CLIP-style alignment misses, which is the stated reason for large gains on GO-BP and GO-CC prediction and on cross-modal transformation.","ProtCLIP can serve as a general protein encoder for classification, mutation-effect regression, semantic similarity, knowledge-graph transformation, and protein-protein interaction prediction, replacing task-specific models.","The claimed 75% average improvement on five cross-modal transformation benchmarks suggests protein-text retrieval and drug-target matching become substantially more reliable.","The property-driven sampling scheme offers a concrete recipe for balancing data quality and quantity in other large-scale multimodal pre-training settings with noisy labels."],"supporting_citations":[{"why":"Supplies the UniProt source of protein sequences and functional annotations used to build ProtAnno.","marker":"Consortium 2019"},{"why":"Describes Swiss-Prot and TrEMBL, the two databases whose different annotation confidence levels motivate the property-driven sampling strategy.","marker":"Bairoch and Apweiler 2000"},{"why":"Provides the CLIP global contrastive loss and language-supervised pre-training paradigm that ProtCLIP extends with segment-level objectives.","marker":"Radford et al. 2021"},{"why":"ProtST is the main multi-modal protein-text baseline and the prior claim that data quality outweighs data quantity, which ProtCLIP directly challenges.","marker":"Xu et al. 2023"},{"why":"Provides ESM-2, the pre-trained protein language model used to initialize ProtCLIP's protein encoder.","marker":"Lin et al. 2023"},{"why":"Provides PubMed-BERT, the biomedical text encoder used for biotext descriptions.","marker":"Gu et al. 2021"},{"why":"Motivates confidence-based use of noisy data in the sampling strategy.","marker":"Berthelot et al. 2019"},{"why":"Motivates the confidence-driven handling of noisy labels behind the property-driven sampling distribution.","marker":"Li, Socher, and Hoi 2020"},{"why":"BioBridge is the strongest baseline in cross-modal transformation, semantic similarity, and protein-protein interaction comparisons.","marker":"Wang et al. 2024"}],"fun_headline_variants":["ProtCLIP beats 22 benchmarks with function-aware segments","Protein-text model ProtCLIP lifts GO and retrieval by up to 75%","Function-informed ProtCLIP sets SOTA on all 22 protein tasks","Segment-wise training helps ProtCLIP top 22 benchmarks","ProtCLIP's property-driven sampling powers cross-modal gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never reports removing from its 251.5-million-pair ProtAnno-D pre-training corpus proteins that are identical or highly similar to proteins in the downstream test sets, so the reported gains could come from memorizing those test proteins instead of learning transferable function understanding.","fun_headline_variants_meta":{"raw":{"variants":["ProtCLIP beats 22 benchmarks with function-aware segments","Protein-text model ProtCLIP lifts GO and retrieval by up to 75%","Function-informed ProtCLIP sets SOTA on all 22 protein tasks","Segment-wise training helps ProtCLIP top 22 benchmarks","ProtCLIP's property-driven sampling powers cross-modal gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1370,"prompt_tokens":975,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":591,"tokens_out":395,"duration_ms":4677,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:39:14.742196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether ProtCLIP's advantage survives deduplication: exclude from ProtAnno-D every protein with, say, 30% or higher sequence identity to any test protein in GO, DeepLoc, TAPE, FLIP, SHS27K, SHS148K, and STRING, retrain, and compare benchmark scores; if the reported SOTA margins mostly disappear, the gains come from memorizing near-duplicates rather than from transferable function understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the UniProt source of protein sequences and functional annotations used to build ProtAnno."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes Swiss-Prot and TrEMBL, the two databases whose different annotation confidence levels motivate the property-driven sampling strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ESM-2, the pre-trained protein language model used to initialize ProtCLIP's protein encoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates confidence-based use of noisy data in the sampling strategy."},{"cited_title":"N.; Rangwala, H.; and Anubhai, R","cited_arxiv_id":null,"evidence_quote":"BioBridge is the strongest baseline in cross-modal transformation, semantic similarity, and protein-protein interaction comparisons."}],"review_version":1}