Pith. sign in

REVIEW 5 major objections 5 minor 53 references

ProtCLIP: Function-Informed Protein Multi-Modal Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ProtCLIP claims that a function-informed, segment-aware protein-text pre-training scheme beats all compared baselines on every one of 22 protein benchmarks.

desk verdict A plausible protein-text pre-training paper with a genuinely large new dataset and two sensible new objectives, but the headline SOTA numbers are unverifiable as written due to missing artifacts and a serious benchmark-leakage risk. read the letter →

arxiv 2412.20014 v1 pith:CSMMBWOC submitted 2024-12-28 cs.LG cs.AIq-bio.BM

Hanjing Zhou , Mingze Yin , Wei Wu , Mingyang Li , Kun Fu , Jintai Chen , Jian Wu , Zheng Wang This is my paper · ORCID
classification cs.LGcs.AIq-bio.BM
keywords proteinfoundationmodelmulti-modalpre-trainingprotein-textalignmentfunctionpredictioncontrastivelearningGeneOntologylanguagenoisy-labelsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ProtCLIP aims to make protein foundation models work the way CLIP-style image-text models do, by aligning protein sequences with written functional descriptions. The paper's claim is that previous attempts fell short for two fixable reasons: they underused large, noisy, machine-annotated protein-text data, and they aligned only whole sequences and whole texts, ignoring the local segments that actually determine protein function. ProtCLIP addresses both with a 251.5-million-pair dataset (ProtAnno-D) sampled by a property-driven rule, and with two segment-level objectives that reconstruct masked static segments and align property-grouped dynamic segments to text. On 22 benchmarks spanning function and localization classification, mutation effect prediction, cross-modal transformation, semantic similarity, and protein-protein interaction, the paper reports best-reported results, including 59.9% and 39.7% improvements on Gene Ontology cellular-component and biological-process prediction. If correct, this is a broadly better protein representation for downstream biology, not just a marginal gain on one task.

What carries the argument

The load-bearing mechanism is a four-term pre-training objective. A global contrastive loss (GC) aligns whole protein sequences with whole biotext descriptions. Biotext-guided Static segment Reconstruction (BSR) masks roughly 15% of the sequence as contiguous static segments and reconstructs them through cross-attention to the biotext. Property-grouped Dynamic segment Alignment (PDA) computes residue-to-property similarity weights, sparsifies them with a threshold $\theta$, forms property-grouped dynamic segments, and contrasts those segments with property prototypes. A protein masked-language-modeling loss (MLM) preserves unimodal knowledge, with weights $\lambda_1+\lambda_2=1$ balancing segment and token reconstruction. On the data side, a property-driven sampling rule selects pre-training pairs with probability proportional to $P \propto C^{-3}\sqrt{R}N$, where $C$ is annotation confidence (lower is more reliable), $R$ is property coverage, and $N$ is cluster size; this is what lets ProtCLIP exploit the noisy 251.5-million-pair ProtAnno-D rather than the smaller curated set.

What would settle it

Check whether ProtCLIP's advantage survives deduplication: exclude from ProtAnno-D every protein with, say, 30% or higher sequence identity to any test protein in GO, DeepLoc, TAPE, FLIP, SHS27K, SHS148K, and STRING, retrain, and compare benchmark scores; if the reported SOTA margins mostly disappear, the gains come from memorizing near-duplicates rather than from transferable function understanding.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that a protein multi-modal foundation model can be built by explicitly modeling the two ways functional regions appear in a sequence. Static functional segments are contiguous runs of residues; dynamic segments are residue sets flexibly selected by their similarity to each property description. ProtCLIP learns both: BSR reconstructs masked static segments using biotext as guidance, and PDA groups residues by property prototypes and aligns those groups to the corresponding property descriptions. Combined with global contrastive alignment and masked language modeling, and trained on noisy large-scale data with property-driven sampling, the model outperforms all compared baselines on all 22 benchmarks. The paper interprets this as evidence that fine-grained, function-informed supervision is what was missing from earlier protein-text pre-training.

Load-bearing premise

The paper never reports removing from its 251.5-million-pair ProtAnno-D pre-training corpus proteins that are identical or highly similar to proteins in the downstream test sets, so the reported gains could come from memorizing those test proteins instead of learning transferable function understanding.

Editorial extensions

If this is right

  • If ProtCLIP's results hold, noise-heavy machine-annotated protein databases can be converted into effective pre-training signal rather than being discarded for quality.
  • The segment-wise objectives inject functionally local information that global CLIP-style alignment misses, which is the stated reason for large gains on GO-BP and GO-CC prediction and on cross-modal transformation.
  • ProtCLIP can serve as a general protein encoder for classification, mutation-effect regression, semantic similarity, knowledge-graph transformation, and protein-protein interaction prediction, replacing task-specific models.
  • The claimed 75% average improvement on five cross-modal transformation benchmarks suggests protein-text retrieval and drug-target matching become substantially more reliable.
  • The property-driven sampling scheme offers a concrete recipe for balancing data quality and quantity in other large-scale multimodal pre-training settings with noisy labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the same two-objective design—reconstruct masked contiguous segments and align threshold-selected dynamic segments to text—is portable to other sequence-plus-annotation domains, such as DNA regulatory elements or RNA motifs, where function also depends on localized sequence features.
  • Our inference: the sampling formula's exponents are not derived from first principles; a systematic sweep over the confidence and coverage exponents would clarify how much of the gain comes from confidence weighting versus coverage weighting, and likely transfer to other noisy multimodal datasets.
  • Our inference: the headline gains should be read as upper bounds until the pre-training and test corpora are checked for sequence-level overlap; if overlap exists, a deduplicated retraining would give the honest transfer numbers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces ProtCLIP, a multi-modal protein-text pre-training framework that aligns protein sequences with functional textual annotations. It contributes a new large-scale paired dataset ProtAnno (with a dense version containing 251.5M pairs), a property-driven sampling strategy to handle noisy machine-annotated data, and a function-informed pre-training paradigm that combines a global contrastive loss, masked reconstruction of static segments (BSR), property-grouped dynamic segment alignment (PDA), and masked language modeling. The authors report state-of-the-art results on 22 benchmarks across five task types: protein classification engineering, mutation effect prediction, cross-modal transformation, semantic similarity inference, and protein-protein interaction prediction. The paper claims remarkable improvements, including 75% average improvement in cross-modal transformation, 59.9% in GO-CC, and 39.7% in GO-BP protein function prediction.

Significance. If the empirical results are reproducible and not inflated by data leakage, ProtCLIP would be a substantial step forward in protein multi-modal representation learning: the dataset scale (251.5M protein-text pairs) is comparable to image-text pre-training corpora, the sampling strategy addresses a real problem of noisy annotations, and the segment-wise objectives aim at fine-grained functional understanding that prior protein-text models such as ProtST lack. However, the current manuscript does not provide code, data, or model weights, and the absence of deduplication analysis against downstream benchmarks leaves the central SOTA claim unverified. The novelty of the method is interesting and the breadth of evaluation is impressive, but the reliability of the headline numbers is the decisive issue.

major comments (5)
  1. [Section 2.1, Appendix A, Tables 2-5] The pre-training corpus ProtAnno-D is built from SwissProt and trEMBL, and the paired text explicitly includes 'FUNCTION', 'SUBCELLULAR LOCATION', and 'SIMILARITY' fields. Downstream benchmarks such as GO/EC, DeepLoc, TAPE, FLIP, SHS27K/SHS148K, and STRING are also derived from UniProt or related protein databases. The manuscript does not describe any identity or homology filtering between ProtAnno-D and the benchmark test sets. If the same or homologous proteins appear in both, the reported gains (e.g., GO-CC +59.9% and GO-BP +39.7% in Table 3) could reflect memorization of benchmark labels during pre-training rather than transferable function understanding. Please quantify the overlap (exact and at homology/family level) between ProtAnno-D and each benchmark test set, and re-evaluate after removing overlapping entries; the central SOTA claim depends on this analysis.
  2. [Section 3.1, Section 3.7, Appendix C.4, Figures 6-7] The key pre-training hyperparameters θ=0.3, λ1=0.7, λ2=0.3 are selected via ablations on downstream benchmarks (Sub, EC, Prot2MF, and Sub again) that are also included in the main reported results. This constitutes test-set tuning for those benchmarks and undermines the claim of consistent SOTA across all 22 benchmarks. The authors should select hyperparameters on held-out validation splits (or a separate pre-training validation set), fix them before evaluating on any test set, and clearly document that the reported test numbers were not used for model or hyperparameter selection.
  3. [Equations (6)-(7)] The PDA objective is underspecified. Equation (6) defines each dynamic segment e_i as a set of thresholded residue vectors, but Equation (7) computes cosine similarity between e_i and the property description embedding a_i, which requires e_i to be a single vector. The paper does not state how e_i is aggregated (e.g., sum, mean, attention pooling) or how gradients are handled through the hard threshold in Equation (5). Please provide the exact vectorization of e_i and the gradient estimation method; without this, the PDA loss and its ablations are not reproducible.
  4. [Equation (7)] The two summed terms in LPDA are written identically. In a standard symmetric InfoNCE loss, the second term should use the text embedding a_i as the query against all protein segment embeddings e_k, while the first term uses e_i as the query against all text embeddings a_k. As written, the loss is not symmetric and may reduce to a doubled one-directional objective. Please correct the equation and verify that the implemented loss matches the intended bidirectional alignment.
  5. [Abstract, Section 2-6] The manuscript provides no code, data, or trained model weights, and the ProtAnno dataset is not released. Since the central contributions are a new pre-training dataset and a new model, the empirical claims cannot be independently checked. Please include an availability statement and, at minimum, release the deduplicated pre-training data and model weights (or a representative subset) along with evaluation scripts, so that the reported benchmark results are verifiable.
minor comments (5)
  1. [Abstract vs Table 3] The abstract reports a 39.7% improvement in GO-BP, but the numbers in Table 3 imply a 65.8% AUPR improvement and a 19.1% Fmax improvement over ProtST-ESM2; please clarify how the 39.7% figure is computed.
  2. [Section 3.4] The description of the cross-modal transformation setup is incomplete; please specify how the transformation module is trained, what negative samples are used, and how the baseline KG methods are configured, so that the comparison is fair.
  3. [Section 3.7, Figure 5] Figure 5 is not referenced clearly in the text and the two curves ('No loss weights assigned' and 'Loss weights λ1=0.7, λ2=0.3') are not labeled with a legend; please add a clear caption, axis labels, and legend.
  4. [Table 2 header] The header contains 'AA' and 'V' as separate column names; these should be merged into 'AAV' to avoid confusion with amino acid abbreviations.
  5. [Appendix C.4, Figure 7] Figure 7 lacks a legend and the y-axis label is partially cut off; also specify the metric (Sub accuracy) in the caption, as the current text says 'Accuracy' without indicating the benchmark.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; ProtCLIP's claims rest on external benchmark comparisons and its pretraining objectives are not reductions of the reported results.

full rationale

The paper's central claim is an empirical SOTA claim evaluated on external benchmarks (GO/EC/DeepLoc, TAPE/FLIP/PEER, PrimeKG-based cross-modal tasks, KeAP similarity, SHS/STRING PPI) against independently pretrained baselines. The pretraining losses (GC, BSR, PDA, MLM) are standard alignment and reconstruction objectives: they define training signals, not fitted predictors of the downstream metrics. No equation in Section 2 reduces a reported benchmark number to an input of the loss; the hyperparameters (lambda1, lambda2, theta) are chosen from ablations on downstream tasks, which is a model-selection concern rather than a constructed equivalence. The 'static/dynamic segments' are modeling constructs, not definitions of the evaluation labels, and the reported results are not derived from them by construction. The only self-citations (Yin et al. 2024, Zhou et al. 2024) appear as background and are not load-bearing; no uniqueness theorem or prior result by the same authors is invoked to force the method. The most serious concern is data contamination: ProtAnno-D is built from SwissProt/trEMBL with no stated deduplication against downstream test sets, so GO/DeepLoc/PPI improvements could be inflated by memorization. That is a validation and reproducibility risk, not a circularity, because the paper's numbers are not equivalent to its inputs by construction.

Assumptions & free parameters 7 free parameters · 5 assumptions · 3 invented entities

The central claim rests on the new dataset and the two new losses, whose hyperparameters are either hand-chosen or tuned on downstream validation. The invented static and dynamic segment concepts lack independent biological evidence, and the paper makes no explicit claim that they correspond to known functional residues. The most consequential assumption is the absence of leakage control between ProtAnno-D and the benchmark evaluation sets.

free parameters (7)
  • lambda1 (BSR loss weight) = 0.7
    Selected by ablation on Sub and Bin benchmarks in Section 3.7; constrained by lambda1 + lambda2 = 1.
  • lambda2 (MLM loss weight) = 0.3
    Complement of lambda1 due to the constraint lambda1 + lambda2 = 1; not independently tuned.
  • theta (PDA similarity threshold) = 0.3
    Selected by ablation on the Sub dataset in Appendix C.4, Figure 7, with a narrow advantage over other values.
  • sampling exponent on confidence C = -3
    Hand-chosen in Eq. (1) to define the property-driven sampling distribution; no derivation or sensitivity analysis is provided.
  • confidence and coverage cutoffs = C=4,5 and R=1/4,2/4 discarded
    Ad hoc filtering thresholds in Section 2.1, justified by data quality reasoning rather than experiment.
  • static segment masking ratio and length = 15% of sequence, length 5-10
    Chosen in Section 2.3 for BSR; standard span masking choices, not tuned.
  • temperature parameters tau1 and tau2 = not reported
    Used in Eqs. (2) and (7); values are not given, which is required for exact reproduction.
assumptions (5)
  • domain assumption Protein function is primarily determined by contiguous or discrete static and dynamic functional segments in 1D sequence.
    Introduced in Section 1 and Section 2.3 as motivation; no biological validation that BSR-masked spans or PDA-aggregated segments correspond to known functional domains.
  • domain assumption UniProt SwissProt and trEMBL textual descriptions are a sufficient and reliable text modality for aligning protein function.
    The entire ProtAnno dataset is built from UniProt text; errors or inconsistencies in these descriptions propagate into training.
  • domain assumption The four selected property objects (protein name, function, subcellular location, similarity) are adequate to capture protein function.
    Appendix A selects these four from UniProt; other properties such as catalytic activity or domains are omitted.
  • standard math Standard InfoNCE and cross-entropy losses are valid for this alignment task.
    Eqs. (2), (3), (7), and (9) use standard losses from Radford et al. and Devlin et al.
  • ad hoc to paper No leakage between ProtAnno-D pre-training data and downstream benchmark test sets.
    The paper does not describe any deduplication or identity filtering when constructing ProtAnno-D from UniProt, while downstream benchmarks such as GO, DeepLoc, TAPE, FLIP, and PPI are derived from the same or overlapping databases.
invented entities (3)
  • Static functional segments
    purpose: Contiguous spans of amino acids masked and reconstructed with biotext guidance in the BSR objective.
    Introduced in Section 2.3. No biological validation, no link to known domains or active sites, and no external handle.
  • Dynamic functional segments
    purpose: Weighted and thresholded aggregations of residues aligned to property descriptions in the PDA objective.
    Introduced in Section 2.3 as flexible functional regions. They are defined by the model's similarity weights and threshold, not by external annotation.
  • Property-grouped prototypes in memory bank
    purpose: Approximate property description sentences for unsupervised aggregation and alignment of dynamic segments.
    Section 2.3. Internal learned representations with no external correspondence or falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProtCLIP: Function-Informed Protein Multi-Modal Learning." pith.science (2026). https://pith.science/paper/CSMMBWOC

@misc{pith2026241220014,
  author       = {Pith},
  title        = {Pith review of: ProtCLIP: Function-Informed Protein Multi-Modal Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CSMMBWOC}},
  note         = {Machine review of arXiv:2412.20014}
}
read the original abstract

Multi-modality pre-training paradigm that aligns protein sequences and biological descriptions has learned general protein representations and achieved promising performance in various downstream applications. However, these works were still unable to replicate the extraordinary success of language-supervised visual foundation models due to the ineffective usage of aligned protein-text paired data and the lack of an effective function-informed pre-training paradigm. To address these issues, this paper curates a large-scale protein-text paired dataset called ProtAnno with a property-driven sampling strategy, and introduces a novel function-informed protein pre-training paradigm. Specifically, the sampling strategy determines selecting probability based on the sample confidence and property coverage, balancing the data quality and data quantity in face of large-scale noisy data. Furthermore, motivated by significance of the protein specific functional mechanism, the proposed paradigm explicitly model protein static and dynamic functional segments by two segment-wise pre-training objectives, injecting fine-grained information in a function-informed manner. Leveraging all these innovations, we develop ProtCLIP, a multi-modality foundation model that comprehensively represents function-aware protein embeddings. On 22 different protein benchmarks within 5 types, including protein functionality classification, mutation effect prediction, cross-modal transformation, semantic similarity inference and protein-protein interaction prediction, our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks, 59.9% in GO-CC and 39.7% in GO-BP protein function prediction. The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.

Figures

Figures reproduced from arXiv: 2412.20014 by the authors.

Figure 1
Figure 1. Data distribution of ProtAnno-S and ProtAnno-D with different property coverage. 2.1 Pre-training data Dataset Curation To enable pre-training of the protein multi-modality foundation model aligning biological and natural language, it is essential to build dataset containing large-scale pairs of protein sequences and textual property descriptions. Our pre-training data is sourced from Swis￾sProt and trEMBL (Bairoch … view at source ↗
Figure 2
Figure 2. Comparison results on 22 downstream benchmarks within five types. ProtCLIP consistently achieves the state-of-the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of ProtCLIP. We curate a large-scale protein-biotext dataset [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An overview of downstream tasks within five types. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Mutual interference between segment-wise recon [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ablation study on loss weights. weights whereas it declines smoothly if the appropriate loss weights are assigned. Therefore, it is essential to find the optimal loss wights. In [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation study on different thresholds in property [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 40 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abdine, H.; Chatzianastasis, M.; Bouyioukos, C.; and Vazirgiannis, M. 2024. Prot2Text : Multimodal protein’s function generation with gnns and transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, 10757--10765

    work page 2024

  4. [4]

    J.; S nderby, C

    Almagro Armenteros, J. J.; S nderby, C. K.; S nderby, S. K.; Nielsen, H.; and Winther, O. 2017. Deeploc : prediction of protein subcellular localization using deep learning. Nature Communications, 33(21): 3387--3395

    work page 2017

  5. [5]

    L.; Kiros, J

    Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016. Layer normalization. arXiv preprint arXiv:1607.06450

    arXiv 2016

  6. [6]

    Bairoch, A.; and Apweiler, R. 2000. The SWISS-PROT protein sequence database and its supplement TrEMBL in 2000. Nucleic Acids Research, 28(1): 45--48

    work page 2000

  7. [7]

    Berthelot, D.; Carlini, N.; Goodfellow, I.; Papernot, N.; Oliver, A.; and Raffel, C. A. 2019. MixMatch : A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems

    work page 2019

  8. [8]

    Chandak, P.; Huang, K.; and Zitnik, M. 2023. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1): 67

    work page 2023

Show all 53 references
  1. [9]

    Chaoyi, W.; Xiaoman, Z.; Ya, Z.; Yanfeng, W.; and Weidi, X. 2023. MedKLIP : Medical Knowledge Enhanced Language-Image Pre-Training for X-ray Diagnosis. In International Conference on Computer Vision, 21315--21326

    2023

  2. [10]

    J.-T.; Zhou, G.; Chen, X.; Zhang, T.; Chang, K.-W.; Zaniolo, C.; and Wang, W

    Chen, M.; Ju, C. J.-T.; Zhou, G.; Chen, X.; Zhang, T.; Chang, K.-W.; Zaniolo, C.; and Wang, W. 2019. Multifaceted protein–protein interaction prediction based on siamese residual rcnn. Bioinformatics, 35(14): i305--i314

    2019

  3. [11]

    Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024. InternVL : Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Computer Vision and Pattern Recognition, 24185--24198

    2024

  4. [12]

    Consortium, U. 2019. Uniprot: a worldwide hub of protein knowledge. Nucleic Acids Research, 47(D1): D506–D515

    2019

  5. [13]

    E.; Wittmann, B

    Dallago, C.; Mou, J.; Johnston, K. E.; Wittmann, B. J.; Bhattacharya, N.; Goldman, S.; Madani, A.; and Yang, K. K. 2021. FLIP : Benchmark tasks in fitness landscape inference for proteins. bioRxiv

    2021

  6. [14]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805

    2019 arXiv

  7. [15]

    Elnaggar, A.; Heinzinger, M.; Dallago, C.; Rehawi, G.; Wang, Y.; Jones, L.; Gibbs, T.; Feher, T.; Angerer, C.; Steinegger, M.; Bhowmik, D.; and Rost, B. 2022. ProtTrans : Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Transactions on Pattern A...

    2022

  8. [16]

    Fuying, W.; Yuyin, Z.; Shujun, W.; Varut, V.; and Lequan, Y. 2022. Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning. In Advances in Neural Information Processing Systems

    2022

  9. [17]

    V.; Joulin, A.; and Misra, I

    Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. Imagebind: One embedding space to bind them all. In Computer Vision and Pattern Recognition, 15180--15190

    2023

  10. [18]

    D.; Kosciolek, T.; Leman, J

    Gligorijevi \'c , V.; Renfrew, P. D.; Kosciolek, T.; Leman, J. K.; Berenberg, D.; Vatanen, T.; Chandler, C.; Taylor, B. C.; Fisk, I. M.; Vlamakis, H.; et al. 2021. Structure-based protein function prediction using graph convolutional networks. Nature Communications, 12(1): 1--14

    2021

  11. [19]

    Gu, Y.; Tinn, R.; Cheng, H.; Lucas, M.; Usuyama, N.; Liu, X.; Naumann, T.; Gao, J.; and Poon, H. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1): 1--23

    2021

  12. [20]

    Hendrycks, D.; and Gimpel, K. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415

    2016 arXiv

  13. [21]

    B.; Baranov, M.; Costello, Z.; Barber, K

    Ingraham, J. B.; Baranov, M.; Costello, Z.; Barber, K. W.; Wang, W.; Ismail, A.; Frappier, V.; Lord, D. M.; Ng-Thow-Hing, C.; Van Vlack, E. R.; et al. 2023. Illuminating protein space with a programmable generative model. Nature, 623(7989): 1070--1078

    2023

  14. [22]

    Ioana, B.; Anastasija, I.; Matthias, B.; Goker, E.; Matko, B.; Christos, K.; Alexey A., G.; Matthias, M.; Charles, B.; Razvan, P.; and Jovana, M. 2024. Improving fine-grained understanding in image-text pre-training. In International Conference on Machine Learning

    2024

  15. [23]

    Junnan, L.; Dongxu, L.; Caiming, X.; and Steven, H. 2022. BLIP : Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In International Conference on Machine Learning

    2022

  16. [24]

    Junnan, L.; Dongxu, L.; Silvio, S.; and Steven, H. 2023. BLIP-2 : Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In International Conference on Machine Learning

    2023

  17. [25]

    Li, C.; Wong, C.; Zhang, S.; Usuyama, N.; Liu, H.; Yang, J.; Naumann, T.; Poon, H.; and Gao, J. 2023. LLaVA-Med : Training a Large Language-and-Vision Assistant for Biomedicine in One Day. In Advances in Neural Information Processing Systems

    2023

  18. [26]

    Li, J.; Socher, R.; and Hoi, S. C. 2020. DivideMix : Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394

    2020 arXiv

  19. [27]

    Lin, Z.; Akin, H.; Rao, R.; Hie, B.; Zhu, Z.; Lu, W.; Smetanin, N.; Verkuil, R.; Kabeli, O.; Shmueli, Y.; et al. 2023. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637): 1123--1130

    2023

  20. [28]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 a . Visual Instruction Tuning. In Advances in Neural Information Processing Systems

    2023

  21. [29]

    Liu, S.; Li, Y.; Li, Z.; Gitter, A.; Zhu, Y.; Lu, J.; Xu, Z.; Nie, W.; Ramanathan, A.; Xiao, C.; Tang, J.; Guo, H.; and Anandkumar, A. 2023 b . A Text-guided Protein Design Framework. arXiv preprint arXiv:2302.04611

    2023 arXiv

  22. [30]

    Lv, G.; Hu, Z.; Bi, Y.; and Zhang, S. 2021. Learning unknown from correlations: graph neural network for inter-novel-protein interaction prediction. arXiv preprint arXiv:2105.06709

    2021 arXiv

  23. [31]

    Pujin, C.; Li, L.; Junyan, L.; and Yijin, H. 2023. PRIOR : Prototype Representation Joint Learning from Medical Images and Reports. In International Conference on Computer Vision, 21304--21314

    2023

  24. [32]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748--8763

    2021

  25. [33]

    Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125

    2022 arXiv

  26. [34]

    Rao, R.; Bhattacharya, N.; Thomas, N.; Duan, Y.; Chen, P.; Canny, J.; Abbeel, P.; and Song, Y. 2019. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems

    2019

  27. [35]

    L.; Ma, J.; and Fergus, R

    Rives, A.; Meier, J.; Sercu, T.; Goyal, S.; Lin, Z.; Liu, J.; Guo, D.; Ott, M.; Zitnick, C. L.; Ma, J.; and Fergus, R. 2021. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sci...

    2021

  28. [36]

    Shanehsazzadeh, A.; Belanger, D.; and Dohan, D. 2020. Is Transfer Learning Necessary for Protein Landscape Prediction? arXiv preprint arXiv:2011.03443

    2020 arXiv

  29. [37]

    Su, B.; Du, D.; Yang, Z.; Zhou, Y.; Li, J.; Rao, A.; Sun, H.; Lu, Z.; and Wen, J. 2022. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.05481

    2022 arXiv

  30. [38]

    Sun, Z.; Deng, Z.-H.; Nie, J.-Y.; and Tang, J. 2019. RotatE : Knowledge Graph Embedding by Relational Rotation in Complex Space. In International Conference on Learning Representations

    2019

  31. [39]

    Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, E.; and Bouchard, G. 2016. Complex embeddings for simple link prediction. In International Conference on Machine Learning

    2016

  32. [40]

    C.; and Dogan, T

    Unsal, S.; Atas, H.; Albayrak, M.; Turhan, K.; Acar, A. C.; and Dogan, T. 2022. Learning functional properties of proteins with language models. Nature Communications, 4(3): 227--245

    2022

  33. [41]

    N.; Rangwala, H.; and Anubhai, R

    Wang, Z.; Wang, Z.; Srinivasan, B.; Ioannidis, V. N.; Rangwala, H.; and Anubhai, R. 2024. BioBridge : Bridging Biomedical Foundation Models via Knowledge Graphs. In International Conference on Learning Representations

    2024

  34. [42]

    E.; Chang, H.; and Zou, J

    Wu, K. E.; Chang, H.; and Zou, J. 2024. ProteinCLIP : enhancing protein language models with natural language. bioRxiv

    2024

  35. [43]

    Xu, M.; Yuan, X.; Miret, S.; and Tang, J. 2023. ProtST : Multi-Modality Learning of Protein Sequences and Biomedical Texts. In International Conference on Machine Learning

    2023

  36. [44]

    Xu, M.; Zhang, Z.; Lu, J.; Zhu, Z.; Zhang, Y.; Chang, M.; Liu, R.; and Tang, J. 2022. PEER : A comprehensive and multi-task benchmark for protein sequence understanding. In Advances in Neural Information Processing Systems

    2022

  37. [45]

    W.; He, X.; Gao, J.; and Deng, L

    Yang, B.; tau Yih, S. W.; He, X.; Gao, J.; and Deng, L. 2015. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations

    2015

  38. [46]

    Yang, J.; Kun, X.; Kun, X.; Liwei, C.; Chao, L.; Jianchao, T.; Quzhe, H.; Bin, C.; Chengru, S.; dai, m.; Di, Z.; Wenwu, O.; Kun, G.; and Yadong, M. 2024. Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization. In International Conference on Learni...

    2024

  39. [47]

    Yao, L.; Huang, R.; Hou, L.; Lu, G.; Niu, M.; Xu, H.; Liang, X.; Li, Z.; Jiang, X.; and Xu, C. 2021. FILIP : Fine-grained Interactive Language-Image Pre-Training. arXiv preprint arXiv:2111.07783

    2021 arXiv

  40. [48]

    Yin, M.; Zhou, H.; Zhu, Y.; Lin, M.; Wu, Y.; Wu, J.; Xu, H.; Hsieh, C.-Y.; Hou, T.; Chen, J.; and Wu, J. 2024. Multi-Modal CLIP-Informed Protein Editing. Health Data Science, 4: 0211

    2024

  41. [49]

    Yuxiao, C.; Jianbo, Y.; Yu, T.; Shijie, G.; Xinyu, L.; Ding, Z.; Dimitris N., M.; and Hongxia, Y. 2023. Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens. In Computer Vision and Pattern Recognition, 15095--15104

    2023

  42. [50]

    Zhang, N.; Bi, Z.; Liang, X.; Cheng, S.; Hong, H.; Deng, S.; Zhang, Q.; Lian, J.; and Chen, H. 2022. OntoProtein : Protein Pretraining With Gene Ontology Embedding. In International Conference on Learning Representations

    2022

  43. [51]

    Zhang, X.; Wu, C.; Zhang, Y.; WeidiXie; and Wang, Y. 2023. Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications, 14(1): 4542

    2023

  44. [52]

    Zhou, H.; Yin, M.; Chen, J.; Chen, D.; and Wu, J. 2024. Group-On : Boosting One-Shot Segmentation with Supportive Query. arXiv preprint arXiv:2404.11871

    2024 arXiv

  45. [53]

    Zhou, H.-Y.; Fu, Y.; Zhang, Z.; Cheng, B.; and Yu, Y. 2023. Protein representation learning via knowledge enhanced primary structure reasoning. In International Conference on Learning Representations

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.