Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations

Ahmed Elnaggar; Hazem Alsamkary; Mohamed Elkerdawy; Mohamed Elshaffei

REVIEW 4 major objections 6 minor 2 references

Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Protein language models trained jointly on masked denoising and sequence completion learn richer representations than single-task models.

desk verdict Useful model release and an interesting pretraining recipe, but the causal claim is untested and the paper overstates its own results. read the letter →

arxiv 2505.20052 v1 pith:OTNLGW4V submitted 2025-05-26 cs.LG q-bio.QM

Hazem Alsamkary , Mohamed Elshaffei , Mohamed Elkerdawy , Ahmed Elnaggar This is my paper · ORCID

classification cs.LGq-bio.QM

keywords Ankh3proteinlanguagemodelmulti-taskpretrainingmaskedmodelingsequencecompletionsecondarystructurepredictioncontactfitness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ankh3 asks whether a protein language model trained on more than one self-supervised objective can learn richer representations from protein sequences alone. The paper argues that it can: jointly optimizing masked language modeling at 15%, 20%, and 30% masking with a sequence-completion task that predicts the second half of a protein from the first half produces embeddings that outperform the earlier single-task models Ankh and ESM2 on secondary structure prediction, fluorescence, GB1 fitness, and contact prediction. Against ESM3, which was pretrained with structure tokens, Ankh3 is competitive on secondary structure and contact tasks and substantially better on fluorescence and GB1 fitness, tasks neither model saw during pretraining. If this is right, multi-task pretraining is a route to stronger protein representations without adding data modalities.

What carries the argument

The load-bearing mechanism is a two-task pretraining objective based on the UL2 scheme, implemented in an encoder-decoder T5 architecture. The first task is masked language modeling in which tokens are masked at 15%, 20%, or 30% rates, sampled uniformly per step, and the model is trained to reconstruct the corrupted spans. The second task is sequence completion, in which the encoder reads the first half of a protein sequence and the decoder must generate the remaining half. Each sequence is randomly assigned to one of the two tasks every time it is sampled, so the same protein is seen under both denoising and generation objectives over the course of training. At evaluation, an [NLU] or [S2S] token is prepended, and the frozen backbone representation is pooled and fed to a fixed ConvBERT head.

What would settle it

Re-run Ankh and ESM2 through the exact ConvBERT evaluation pipeline used for Ankh3, or train an Ankh3-sized model with only a single masking probability and no completion task. If the Table 5 gaps shrink to noise or the single-task model matches Ankh3, the multi-task claim is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that combining variable-rate masked denoising with completion of the missing half of a sequence produces a protein language model whose frozen representations transfer better than those of prior sequence-only protein language models. The evidence is a set of benchmarks in which Ankh3-Large and Ankh3-XL beat the published Ankh and ESM2 scores on every task tested, and in which Ankh3-XL outperforms the multimodal ESM3 on fluorescence and GB1 fitness while matching it on most secondary-structure and contact entries. The authors read the scaling behavior as supporting the same conclusion: Ankh3-Large to Ankh3-XL improves on both objectives, whereas the earlier ProtT5 line plateaued when scaled, so the gains are attributed to the multi-task objective and variable masking rather than to model size alone.

Load-bearing premise

The load-bearing premise is that the Ankh and ESM2 scores quoted from the earlier Ankh paper are directly comparable to the new Ankh3 scores, even though they come from a different downstream evaluation setup, and that the performance gap is caused by the pretraining objective rather than by model scale or other training changes.

Editorial extensions

If this is right

If the central claim is right, sequence-only protein language models can be improved by adding a second self-supervised task instead of adding structure or evolution data.
The mixture of 15%, 20%, and 30% masking implies that no single masking rate dominates across downstream tasks, making variable denoising a reasonable default for future pretraining.
Because Ankh3-XL improves over Ankh3-Large on both objectives, the multi-task setup appears to give larger models useful capacity to exploit, unlike the ProtT5 scaling plateau the paper cites.
The observation that [NLU] and [S2S] tokens lead to different rankings across tasks means users may need to select the task token per application.
Strong fluorescence and GB1 fitness results suggest the multi-task representation transfers to fitness landscapes that were not part of pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not isolate how much of the gain comes from multi-task training versus simply varying the masking probability; a same-scale ablation of single-rate MLM, multi-rate MLM, and completion alone would settle that.
The large seed-42 variance reported for ESM3 suggests that single-seed comparisons with strong multimodal baselines can be misleading, so more seeds would make the competitive claim sturdier.
If sequence completion is the active ingredient, variable completion percentages and longer contexts are a natural next experiment that could improve both generation and embedding quality.
The same two-task recipe could transfer to other biological sequence domains, such as RNA or antibody repertoires, where denoising alone has been the default pretraining objective.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, what's worth taking from this paper: they train two T5-style protein models with a multi-task objective borrowed from UL2—MLM at three masking rates plus 50% sequence completion—and release the weights. That's a concrete, reproducible resource for the protein LM community. Training details are documented well, they report three seeds, and the comparison to ESM3, while noisy, is a reasonable thing to include. The observation that S2S helps on classification tasks like GB1/fluorescence while NLU helps on structure prediction is a useful empirical lead. The citation pattern is also fine; prior work, especially UL2 and the Ankh line, is credited properly.

The soft spots are real and fairly large. The central claim in the abstract and Section 4—that multi-task pretraining produces richer, more generalizable representations—is never actually tested. There is no same-scale, same-data control trained with MLM only or completion only. The scaling argument in Section 4 does not substitute for an ablation because ProtT5 differs in data, architecture, and setup. On top of that, the baselines for Ankh and ESM2 are lifted from the original Ankh paper, so they come from a different evaluation pipeline (different head, pooling, and training details). And the text says 'Ankh3 performed better in all tasks' against sequence-only models, but their own Table 5 shows Ankh3-L NLU below Ankh Base/Large on multiple CASP structure and contact benchmarks. That is not a minor wording issue; it is the headline result.

The ESM3 comparison also deserves a caution flag: they rely on one seed (42) producing very high variance to claim competitiveness on contact prediction. That is within the data they report, so it is not cherry-picking exactly, but the conclusion is fragile. No evaluation code is released either, which makes the baseline comparison hard to reproduce.

What the paper does establish: a specific training recipe at 1.9B and 5.7B scale yields competitive or better numbers than ESM2 and Ankh on several tasks, and the weights are public. That is worth having. But it does not establish that the multi-task objective is the cause.

For peer review: yes, send it out. The release is serious, the idea is timely, and the community would benefit from a version where the causal claim is either backed by a proper ablation or downgraded to a descriptive statement. The authors should add at least one same-scale single-task control, rerun the Ankh/ESM2 baselines in their own pipeline, fix the 'all tasks' sentence, and release evaluation code. I would engage with a revised version.

Referee Report

4 major / 6 minor

Summary. The paper introduces Ankh3, an encoder-decoder protein language model pre-trained on UniRef50 with a multi-task objective combining masked language modeling at three masking probabilities (15%, 20%, 30%) and a 50% sequence-completion task. Two model sizes are trained (Ankh3-Large and Ankh3-XL), with architectures based on T5 and training details fully specified. The models are evaluated on secondary structure prediction (CASP12, CASP14), fluorescence (TAPE), GB1 fitness (FLIP), and contact prediction (ProteinNet, CASP14), reporting three-seed averages for NLU and S2S token prompts. The authors claim that this multi-task pre-training yields richer, more generalizable protein representations from sequence-only input and that Ankh3 outperforms previous sequence-only PLMs such as Ankh and ESM2, while being competitive with ESM3 on structure-related tasks.

Significance. If the causal claim were established, this would be a practically valuable result: it would show that a purely sequence-based multi-task objective can improve protein representations without adding structural or other modalities, and the paper openly releases model weights, hyperparameters, and dataset links. The three-seed averaging with standard deviations for Ankh3 is good experimental hygiene, and the comparison against ESM3, despite its variability, is informative. However, the central attribution of the reported gains to the multi-task objective is not identified in any controlled experiment, and several internal results contradict the unqualified claim that Ankh3 performs better in all tasks. The manuscript therefore documents a promising model and a plausible hypothesis, but the evidence as presented does not yet support the paper's headline conclusion.

major comments (4)

[§3.1, Table 5; §4] The paper's central causal claim—that jointly optimizing MLM at multiple masking probabilities and sequence completion yields richer representations—is not supported by a controlled experiment. Ankh3 is never compared with a same-architecture, same-data, same-training-budget model trained on MLM alone or on completion alone; every comparison in Table 5 changes at least one of architecture, scale, data, or downstream pipeline. The ProtT5 scaling comparison invoked in §4 cannot serve as a control because ProtT5 differs in data, architecture, and training setup. Please train at least an Ankh3-Large MLM-only baseline and an Ankh3-Large completion-only baseline, matching the multi-task run in data, optimizer, and step count, and report them in Table 5.
[§3.1, Table 5 footnote] The Ankh and ESM2 baseline numbers in Table 5 are "sourced from the Ankh paper" and were produced with a different downstream evaluation pipeline than the ConvBERT head with global average pooling described in §3 and Table 4. Because downstream head and pooling choices can change results substantially, differences in Table 5 may reflect the evaluation pipeline rather than the pretraining objective. Please rerun the sequence-only baselines (at minimum Ankh Base/Large and ESM2-650M) through the identical evaluation pipeline used for Ankh3, and report those numbers alongside.
[Table 5; Abstract; §4] Several statements in the Abstract and §4 ("Ankh3 performed better in all tasks", "multi-task pre-training demonstrated... improved performance in downstream tasks") are contradicted by the paper's own Table 5. Ankh3-L NLU scores 78.03 on CASP12 SSP-3 versus 83.59 for Ankh Large, 69.42 versus 73.49 on ProteinNet P@L/5, and 26.65 versus 29.91 on CASP14 P@L/5. The benefit is therefore task- and scale-dependent, and the unqualified claims should be revised and, where possible, accompanied by significance tests across seeds.
[§4] The sentence in §4 attributing the performance boost to "the addition of multiple masking probabilities and sequence completion" as the "main contributors" is a causal attribution that the experiments do not establish, because no ablation isolates either factor and the comparison to ProtT5 changes multiple variables. Please remove this attribution or rephrase it as a hypothesis to be tested in future work.

minor comments (6)

[§1] The line "encoder-decoder models ()" contains an empty citation; please add the intended reference.
[Table 1] With embedding dim 1536 and 16 heads, the per-head dimension is 96, not the listed KV dim 64; for the XL model it would be 80. Please clarify whether KV dim denotes a different quantity (e.g., a grouped-query setting) or correct the table.
[Table 5] The row label "ProteinNet (L/1)" should be "P@L" to match the text's definition of Precision@L; as printed it is easy to misread as a length-1 normalization.
[§3.1] The text says "the average performance of each model with both [NLU] and [S2S] tokens" but Table 5 reports separate columns; please clarify whether the average is over seeds only, with tokens shown separately, or whether the two columns themselves are averaged.
[§4] The ProtT5 numbers quoted in §4 (84.4%, 79.2%, 81.4%) are not presented in a table or directly sourced in this manuscript; please add a citation or reproduce the relevant source values for verifiability.
[§2.5] For reproducibility of the completion task, please specify how the 50% split is performed at the token level, how odd-length sequences are handled, and how sentinel tokens are introduced in the completion task.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Ankh3 is an empirical model-building paper whose downstream benchmark numbers are measured against external tasks, and no prediction or derivation reduces to a fitted input or self-citation by construction.

full rationale

The paper's central claim is an empirical outcome: jointly optimizing masked language modeling at 15%, 20%, and 30% masking with a 50% sequence completion task yields the downstream scores reported in Table 5. These scores are measured results, not outputs derived from model equations or from fitted parameters renamed as predictions. The self-referential elements are limited to (i) using the authors' prior Ankh/ProtTrans training data and T5-style architecture, and (ii) comparing against prior Ankh and ProtT5 numbers sourced from Elnaggar et al. Those citations are not load-bearing in a circular sense: the prior Ankh results are independent empirical baselines, and the ProtT5 scaling experiments are external observations rather than assumptions that already contain the conclusion. The paper's inference that the multi-task objective caused the performance gains is under-supported because no same-scale single-task ablation is run and the external baselines use a different evaluation pipeline; however, this is a comparability and confounding concern, not circularity. No equation, definition, or uniqueness claim makes the conclusion equivalent to its inputs. Accordingly, no circular step is identified, and the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no invented entities. The load-bearing hand-chosen hyperparameters are the masking probability set (15%, 20%, 30%) and the 50% completion fraction; both are design decisions rather than fitted constants, and neither is ablated. The evaluation relies on standard benchmark assumptions from prior literature, including UniRef50 as a pretraining corpus, frozen-backbone probing with a ConvBERT head, and DSSP/contact definitions as ground truth.

free parameters (2)

masking probabilities set = 15%, 20%, 30%, sampled uniformly
Hand-chosen based on prior Ankh experiments that showed differing optimal denoising probabilities across tasks; not optimized for the reported benchmarks but a design choice central to the multi-task claim.
completion fraction = 50%
The sequence is split in half; chosen by hand and not varied ('variable completion percentages... was not experimented') due to compute limits. This is a free design choice.

assumptions (3)

domain assumption UniRef50 is an appropriate pretraining corpus for protein language models
The paper relies on UniRef50 clustering to reduce redundancy; this is standard practice in the field and not independently validated here.
domain assumption Frozen backbone representations with a ConvBERT head are a valid measure of representation quality
All downstream evaluations use this protocol; if this protocol is flawed, the cross-model comparisons are invalid.
domain assumption DSSP-derived secondary structure labels and C-alpha-C-alpha distance less than 8 Angstroms contact definitions are valid ground truth
Standard benchmark definitions taken from prior literature; the paper does not validate them.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations." pith.science (2026). https://pith.science/paper/OTNLGW4V

@misc{pith2026250520052,
  author       = {Pith},
  title        = {Pith review of: Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTNLGW4V}},
  note         = {Machine review of arXiv:2505.20052}
}

read the original abstract

Protein language models (PLMs) have emerged as powerful tools to detect complex patterns of protein sequences. However, the capability of PLMs to fully capture information on protein sequences might be limited by focusing on single pre-training tasks. Although adding data modalities or supervised objectives can improve the performance of PLMs, pre-training often remains focused on denoising corrupted sequences. To push the boundaries of PLMs, our research investigated a multi-task pre-training strategy. We developed Ankh3, a model jointly optimized on two objectives: masked language modeling with multiple masking probabilities and protein sequence completion relying only on protein sequences as input. This multi-task pre-training demonstrated that PLMs can learn richer and more generalizable representations solely from protein sequences. The results demonstrated improved performance in downstream tasks, such as secondary structure prediction, fluorescence, GB1 fitness, and contact prediction. The integration of multiple tasks gave the model a more comprehensive understanding of protein properties, leading to more robust and accurate predictions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

[1]

A., Tam` o, G

Abriata, L. A., Tam` o, G. E., Monastyrskyy, B., Kryshtafovych, A., & Dal Peraro, M. (2018). Assess- ment of hard target modeling in casp12 reveals an emerging role of alignment-based contact prediction methods. Proteins: Structure, Function, and Bioinformatics, 86(S1), 97–112. https: //doi.org/https://doi.org/10.1002/prot.25423 6 AlQuraishi, M. (2019). P...

arXiv 2018
[9856]

S., Anand, N., Eguchi, R

Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P.-S., & Socher, R. (2020). Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497 . Nijkamp, E., Ruffolo, J., Weinstein, E. N., Naik, N., & Madani, A. (2022). Progen2: Exploring the boundaries of protein language models. https://arxiv.org/abs/2206...

arXiv 2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

A., Tam` o, G

Abriata, L. A., Tam` o, G. E., Monastyrskyy, B., Kryshtafovych, A., & Dal Peraro, M. (2018). Assess- ment of hard target modeling in casp12 reveals an emerging role of alignment-based contact prediction methods. Proteins: Structure, Function, and Bioinformatics, 86(S1), 97–112. https: //doi.org/https://doi.org/10.1002/prot.25423 6 AlQuraishi, M. (2019). P...

arXiv 2018

[2] [9856]

S., Anand, N., Eguchi, R

Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P.-S., & Socher, R. (2020). Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497 . Nijkamp, E., Ruffolo, J., Weinstein, E. N., Naik, N., & Madani, A. (2022). Progen2: Exploring the boundaries of protein language models. https://arxiv.org/abs/2206...

arXiv 2020