{"id":"f521c611-be12-49d8-9a71-fd58a7e88e91","arxiv_id":"2411.15215","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"S2ALM pretrains a 650M-parameter antibody language model on sequence and Foldseek 3Di structure tokens, reporting modest state-of-the-art gains across six antibody benchmarks.","lead":"A new antibody AI model, S2ALM, was trained on both the amino-acid sequence and a simplified 3D shape code from 75 million proteins and antibodies. It reports small but consistent gains over previous antibody models on binding, maturation, and design tasks, but has not yet released code or data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed SOTA advantage is not statistically supported: Table 1 lacks error bars, affinity gains overlap baselines, and no matched sequence-only control isolates the structure benefit.","rationale":"The reader's CONDITIONAL verdict is appropriate, but the weakest assumption is not primarily the information loss of Foldseek 3Di tokens. Foldseek's 3Di encoding is an established, if lossy, structural representation used by other protein models, and the paper's ablation shows that the structure-derived objectives contribute to downstream performance. The more immediate, load-bearing weakness is that the empirical evidence for the headline SOTA claim is statistically inconclusive: Table 1 has no error bars, Table 2's confidence intervals overlap for several key metrics, and no matched sequence-only control is provided to attribute gains to structural information rather than model scale or auxiliary training objectives. These gaps directly affect the central claim of outperforming baselines and setting new SOTA. They do not require rejecting the work; rather, they justify the CONDITIONAL verdict and the need for released artifacts, error bars, and controlled ablations. The Data Availability copy-paste reinforces reproducibility concerns but is secondary to the statistical and control issues.","tokens_in":19224,"tokens_out":3509,"duration_ms":44036,"concrete_test":"Retrain the downstream tasks in Tables 1 and 2 with at least 5 random seeds and report mean±std and pairwise significance tests; additionally, train a 650M-parameter ESM-2-style model on the same 75M sequences with no structural tokens (sequence-only control) and evaluate on the identical benchmarks. If the S2ALM improvements over the best baseline are within one standard deviation, or if the sequence-only control matches S2ALM, the SOTA claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that S2ALM 'outperforms well-established and renowned baselines and sets new state-of-the-art performance.' The reported evidence does not securely establish this. In Table 1, no error bars or significance tests are given; the margins over the best baseline are small (e.g., antigen binding AUC 0.931 vs 0.922; maturation ACC 0.588 vs 0.581; paratope AUC 0.898 vs 0.887), and without variance estimates these could easily be noise. In Table 2, error bars are provided, and they overlap for the key comparisons: 14H Pearson is 0.650±0.013 for S2ALM vs 0.642±0.012 for A2binder; BioMap Spearman is 0.749±0.028 vs 0.746±0.025. Thus the 'new SOTA' on binding affinity is not distinguishable from the runner-up at the reported precision. Furthermore, the ablation in Table S2 does not isolate the contribution of structure: 'w/o SSM&CLR' still includes stage-I MLM on 3Di tokens and stage-II 1D MLM, so it removes only the two auxiliary objectives, not the structural input modality. No comparison is made to a same-architecture, same-data sequence-only model of 650M parameters. Therefore the paper does not demonstrate that structure-aware pretraining, rather than scale or auxiliary objectives, drives the improvements. The Data Availability section is an obvious copy-paste from an unrelated CLIP protein-editing paper (mentioning 'biotext-protein paired dataset' and 'CLIP-informed protein editing'), which further weakens confidence in the reproducibility promise, but the core scientific concern is the statistical and control deficiency in the headline results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents S2ALM, a 650M-parameter antibody language model pre-trained with a two-stage hierarchical scheme on 75 million sequences and 11.7 million structures. Structures are discretized with Foldseek into 3Di tokens. Stage I performs masked language modeling on 1D and 3Di sequences from general proteins; stage II adds two antibody-specific objectives, Sequence-Structure Matching (SSM) and Cross-Level Reconstruction (CLR). The model is evaluated on antigen-binding classification, B-cell maturation classification, paratope prediction, binding-affinity regression, and CDR-H3 design, reporting state-of-the-art results against sequence-only antibody and protein language models. The paper also presents t-SNE analyses, attention-based structural interpretability, and AlphaFold3-based structural evaluation of designed CDRs.","tokens_in":19485,"tokens_out":5286,"duration_ms":52139,"significance":"If the reported results are statistically robust, S2ALM would be a substantial advance: it is among the first antibody-specific foundation models to incorporate structure at scale, and the two-stage design with the SSM/CLR objectives is a reasonable and potentially reusable approach. The paper provides a broad set of downstream benchmarks, an ablation showing that both objectives contribute, and interpretability analyses; these are strengths that should be credited. However, the lack of error bars in Table 1, overlapping error bars in Table 2, and the absence of a matched sequence-only control for the structure stream mean that the central 'new state-of-the-art' claim is not yet firmly established. The contribution is promising but conditional on stronger statistical and ablative evidence.","major_comments":[{"comment":"Table 1 reports no error bars or significance tests, and the margins over the best baseline are small (e.g., antigen binding AUC 0.931 vs 0.922, maturation ACC 0.588 vs 0.581, paratope AUC 0.898 vs 0.887). Without variance estimates or paired significance tests, the abstract's claim of 'new state-of-the-art performance' and the statement in Section 3.1 that S2ALM 'outperforms well-established and renowned baselines' are not statistically supported.","section":"Table 1; §3.4–3.6"},{"comment":"In Table 2 the provided error bars overlap for key comparisons: 14H Pearson is 0.650±0.013 for S2ALM versus 0.642±0.012 for A2binder, and BioMap Spearman is 0.749±0.028 versus 0.746±0.025. The text states that S2ALM 'significantly surpasses all baselines' except BioMap Spearman, but at the reported precision the differences are within one standard error. Please report the number of runs, the meaning of the parenthetical values, and appropriate significance tests or confidence intervals.","section":"Table 2; §3.7"},{"comment":"The ablation in Table S2 does not isolate the contribution of structural input. The 'w/o SSM&CLR' condition still includes stage-I MLM on 3Di tokens and stage-II 1D MLM, so it removes only the two auxiliary objectives, not the structural modality. Without a same-architecture, same-data sequence-only 650M-parameter control, the paper cannot support the claim that structure-aware pretraining (rather than model scale, data volume, or the auxiliary objectives) drives the observed gains. Please add such a control or clearly qualify the claim.","section":"Table S2; §2.3"}],"minor_comments":[{"comment":"The text states that 216,437,989 unique antibody sequences were obtained but then says 10 million sequences were randomly selected for training; please clarify why only a subset was used and whether the random selection was stratified or otherwise controlled.","section":"§2.1.2"},{"comment":"The sentence 'S2ALM significantly surpasses all baselines, except for the BioMap dataset, where it achieves the same Spearman correlation as A2binder' is internally inconsistent; please rephrase to distinguish 'best' from 'significantly better'.","section":"§3.7"},{"comment":"The Data Availability statement mentions 'biotext-protein paired dataset' and 'CLIP-informed protein editing', which do not correspond to the antibody data and models in this paper and appear to be copied from another manuscript; please replace with an accurate statement for S2ALM.","section":"Data Availability"},{"comment":"The caption refers to 'the 3th head' of the last hidden layer; this should be 'the 3rd head'.","section":"Fig. 4 caption"},{"comment":"The parenthetical values in Table 2 are not defined in the caption or text; please state whether they are standard deviations over random seeds, data splits, or something else.","section":"Table 2 caption"},{"comment":"The structural interpretability analysis relies on a single antibody (STE90-C11, PDB 7B3O); consider quantifying how often high-attention residue pairs coincide with known hydrogen bonds or contacts across a larger set of antibody structures.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The Data Availability section appears to be an accidental copy from the authors' related CLIP protein-editing paper (reference [56]); it should be corrected before publication. This is a reproducibility and integrity issue, but it reads as a template error rather than intentional deception. In addition, the paper does not discuss possible overlap between the OAS-derived pretraining corpus and the benchmark test sets; since OAS contains a large fraction of publicly deposited antibody sequences, the authors should check for sequence identity between their 10M pretraining subset and the test sequences used in Tables 1–2 and the CDR design task. This would address the residual circularity concern raised by the downstream evaluation setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serious pretraining effort and a genuinely new combination—Foldseek 3Di tokens plus an ESM-2 backbone with two structure-focused objectives—but the \"new SOTA\" claim is not supported by the numbers as reported, and the paper as posted cannot be checked. It deserves peer review, not desk rejection, but it needs real revision before the conclusions can stand.\n\nThe genuinely new thing is the hybrid encoder. S2ALM is, to my knowledge, the first antibody-specific language model pretrained jointly on sequences and structure-derived 3Di tokens at this scale, and the two stage-II objectives (SSM and CLR) are a sensible way to force cross-level alignment. The ablation in Table S2 is the most convincing part: removing SSM, CLR, or both consistently hurts performance across antigen binding, maturation, and paratope prediction. The pretraining corpus—75 million sequences and 11.7 million predicted/experimental structures—is also substantial. The t-SNE and attention analyses are fine as sanity checks, nothing more.\n\nThe soft spots are mostly about the headline. Table 1 has no error bars or significance tests, and the margins over the best baseline are small: 0.009 AUC on antigen binding, 0.007 on maturation ACC, 0.011 on paratope AUC. With no variance estimate, those could easily be noise. Table 2 does report errors, but the key comparisons overlap: 14H Pearson is 0.650±0.013 for S2ALM versus 0.642±0.012 for A2binder; BioMap Spearman is 0.749±0.028 versus 0.746±0.025. The paper's claim that S2ALM \"significantly surpasses all baselines\" does not follow from those numbers. The ablation also does not isolate structure: the \"w/o SSM&CLR\" condition still trains on 3Di tokens in stage I, so it removes only the two auxiliary objectives, not the structural input modality. A same-architecture, same-data, sequence-only 650M model is the right control, and it is not there. Finally, the Data Availability section is an obvious copy-paste from a different paper about CLIP-informed protein editing, and no code, weights, or data are released. That alone would force a conditional verdict from me.\n\nIs the central idea sound? I think yes, conditionally. Structure-aware pretraining for antibodies is plausible, the ablation supports the auxiliary objectives, and the problems are in evidence quality and presentation, not in the architecture. But the conclusions currently outrun the statistics and the artifacts.\n\nWho gets value: researchers working on antibody or protein foundation models, and anyone planning a structure-aware pretraining run. This paper deserves a serious referee—send it out—but the review must ask for matched controls, error bars, a decontamination analysis, and a corrected data-availability statement with actual release.","headline":"A real large-scale structure-aware antibody pretraining effort with a plausible core idea, but the SOTA claims outrun the reported statistics and the paper as posted is not reproducible.","tokens_in":20168,"tokens_out":2658,"would_cite":true,"duration_ms":30305,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","92D20"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces S2ALM, a large pre-trained antibody language model that jointly encodes amino-acid sequences and 3D structures via Foldseek's 3Di tokens, and claims state-of-the-art performance across antigen binding prediction, B…","keywords":["antibody language model","pre-training","protein structure","3Di tokens","sequence-structure matching","cross-level reconstruction","antigen binding prediction","CDR design"],"falsifier":"Train S2ALM with the same pipeline but with 3Di sequences randomly shuffled or replaced by a fixed token per residue, so the structural channel carries no real geometry. If performance on antigen binding, paratope, and affinity tasks stays at the reported levels, the structural encoding is not the cause of the gains; if performance drops, the structural information is causative.","tokens_in":18933,"feed_emoji":"🧬","tokens_out":5335,"duration_ms":49626,"temperature":0.7,"pith_summary":"The paper argues that existing antibody language models are limited because they read only the amino-acid sequence, ignoring the 3D structure that actually determines binding. It proposes S2ALM, a large pre-trained model that jointly encodes antibody sequences and structures, using Foldseek's 3Di tokens to turn structure into a sequence-like input. Pre-trained on 75 million sequences and 11.7 million structures, S2ALM claims state-of-the-art results on antigen binding prediction, B cell maturation classification, paratope prediction, binding affinity prediction, and CDR design. The implication, if correct, is that a single general-purpose antibody model can improve both understanding and generation of antibodies for therapeutic development.","feed_headline":"Antibody model that reads 3D structure beats sequence-only rivals","feed_subtitle":"Trained on 75M sequences and 11.7M structures, it tops binding, paratope, affinity, and design tasks.","key_machinery":"The key mechanism is the hybrid token vocabulary of 20 standard amino acids plus Foldseek's 20 3Di tokens, which discretize each residue's local 3D interaction into a single symbol. Foldseek converts a 3D structure into a 1D 3Di sequence, allowing the transformer to process sequence and structure in a unified format. Two pre-training objectives then bind the levels: Sequence-Structure Matching (SSM) is a binary classification of whether a 1D and a 3Di sequence come from the same antibody, and Cross-Level Reconstruction (CLR) masks tokens in one level and reconstructs them from the other. This machinery is what lets a single model absorb both modalities during masked language modeling.","core_discovery":"The central claim is that injecting structural information into antibody pre-training, through a hybrid vocabulary of amino-acid and 3Di tokens, yields a general antibody foundation model that outperforms sequence-only antibody and protein language models. The paper demonstrates this with a two-stage pre-training pipeline: first general protein sequence-structure learning, then antibody-specific multi-level learning with two new objectives, Sequence-Structure Matching and Cross-Level Reconstruction. The resulting representations capture functional specificity, species, and isotype information, and the model sets new state-of-the-art numbers across four understanding tasks and one generation task.","pith_inferences":["The same 3Di-based hybrid pretraining could transfer to other protein families with few experimental structures, such as T-cell receptors or nanobodies, where predicted structures from AlphaFold-like models are plentiful.","As Foldseek's 3Di vocabulary improves or is replaced by finer structural tokens, S2ALM-style models should scale in performance without architectural change; the current 20-state discretization is likely a bottleneck, as the authors acknowledge.","A testable extension is to apply the SSM and CLR objectives to paired sequence-structure data from proteins other than antibodies to see if the structural benefit generalizes beyond the immune system."],"forward_implications":["A single pre-trained model can serve both understanding tasks (binding, maturation, paratope, affinity) and generation (CDR infilling), replacing task-specific antibody models.","Structure-aware representations improve paratope prediction, the token-level task most directly tied to 3D binding interfaces, as shown in the paper's ablation where removing SSM hurts paratope prediction most.","The two-stage curriculum (general protein then antibody-specific) lets scarce antibody structure data be supplemented by abundant protein structure data without losing antibody specificity.","With 3Di tokens, existing sequence-model architectures can be adapted to structure without geometric deep nets, lowering the barrier for structure-aware antibody models."],"supporting_citations":[{"why":"Supplies the 3Di tokenization method that converts 3D structure into a 1D pseudo-sequence, the core encoding technique of the model.","marker":"[22]"},{"why":"Provides the ESM-2 architecture the model is built on and serves as a strong sequence-only baseline for comparison.","marker":"[29]"},{"why":"Provides the Observed Antibody Space database, the source of the 10 million pre-training antibody sequences.","marker":"[23]"},{"why":"Supplies 10 million computationally predicted protein structures from AlphaFoldDB used in the general sequence-structure learning stage.","marker":"[19]"},{"why":"Supplies 1.4 million computationally predicted antibody structures from IgFold for the antibody-specific stage.","marker":"[21]"},{"why":"Supplies 150 thousand computationally predicted antibody structures from ABodyBuilder2 for the antibody-specific stage.","marker":"[20]"},{"why":"Provides a key sequence-only antibody language model baseline and the paratope prediction dataset used in one of the two paratope benchmarks.","marker":"[13]"},{"why":"Provides another sequence-only antibody language model baseline and supplies the datasets for antigen binding, maturation, and paratope tasks.","marker":"[16]"}],"fun_headline_variants":["Fusing sequence and structure tops antibody benchmarks","Structure-aware antibody model outperforms sequence-only rivals","75M sequences plus 11.7M structures yield top antibody AI","Antibody model learns from both 1D and 3D to excel at tasks","Sequence-structure pretraining sets new antibody AI records"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on Foldseek's 20-state 3Di tokens preserving the structural details that determine antibody binding; if those discretized tokens discard the geometry that matters, the structural advantage claimed over sequence-only models would not be genuine.","fun_headline_variants_meta":{"raw":{"variants":["Fusing sequence and structure tops antibody benchmarks","Structure-aware antibody model outperforms sequence-only rivals","75M sequences plus 11.7M structures yield top antibody AI","Antibody model learns from both 1D and 3D to excel at tasks","Sequence-structure pretraining sets new antibody AI records"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":3010,"prompt_tokens":963,"completion_tokens":2047,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1962}},"tokens_in":579,"tokens_out":2047,"duration_ms":16205,"temperature":1.0,"reasoning_tokens":1962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:31:45.960090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train S2ALM with the same pipeline but with 3Di sequences randomly shuffled or replaced by a fixed token per residue, so the structural channel carries no real geometry. If performance on antigen binding, paratope, and affinity tasks stays at the reported levels, the structural encoding is not the cause of the gains; if performance drops, the structural information is causative.","supporting_citations":[{"cited_title":"Fast and accurate protein structure search with Foldseek","cited_arxiv_id":null,"evidence_quote":"Supplies the 3Di tokenization method that converts 3D structure into a 1D pseudo-sequence, the core encoding technique of the model."},{"cited_title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","cited_arxiv_id":null,"evidence_quote":"Provides the ESM-2 architecture the model is built on and serves as a strong sequence-only baseline for comparison."},{"cited_title":"Observed Antibody Space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences","cited_arxiv_id":null,"evidence_quote":"Provides the Observed Antibody Space database, the source of the 10 million pre-training antibody sequences."},{"cited_title":"AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models","cited_arxiv_id":null,"evidence_quote":"Supplies 10 million computationally predicted protein structures from AlphaFoldDB used in the general sequence-structure learning stage."},{"cited_title":"Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies","cited_arxiv_id":null,"evidence_quote":"Supplies 1.4 million computationally predicted antibody structures from IgFold for the antibody-specific stage."},{"cited_title":"ImmuneBuilder: Deep-Learning models for predicting the structures of immune proteins","cited_arxiv_id":null,"evidence_quote":"Supplies 150 thousand computationally predicted antibody structures from ABodyBuilder2 for the antibody-specific stage."},{"cited_title":"Deciphering the language of antibodies using self-supervised learning","cited_arxiv_id":null,"evidence_quote":"Provides a key sequence-only antibody language model baseline and the paratope prediction dataset used in one of the two paratope benchmarks."},{"cited_title":"On Pre-trained Language Models for Antibody","cited_arxiv_id":null,"evidence_quote":"Provides another sequence-only antibody language model baseline and supplies the datasets for antigen binding, maturation, and paratope tasks."}],"review_version":1}