REVIEW 2 major objections 4 minor 119 references
A 52.6B-token biology corpus more than doubles a fixed 16B model's score on a matched biological eval, with general language nearly intact.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 07:54 UTC pith:MIZPMXWI
load-bearing objection The corpus is a real resource worth building on, but the +0.114 corpus-attribution claim is weakened by an unmatched-compute ablation (54B vs ~106B tokens), so the paper needs revision rather than rejection. the 2 major comments →
TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On its own terms, the discovery is that a consolidated, enriched, instruction-bearing corpus transfers biological knowledge to a general LLM at pre-training scale. Each record pairs an entity's sequence or identifier with a natural-language description and computed properties; cross-domain records chain a protein to its pathway, its binders, and the cellular response they induce. The paired evaluation suite asks for recognition, generation, prediction, and two-hop reasoning across five domains. The authors report the average evaluation score rises from 0.223 to 0.499, and that the portion not explained by web-text annealing is +0.114.
What carries the argument
The central mechanism is a corpus-construction pipeline that renders biological databases in a shared language interface: tagged tokens keep SMILES, protein, DNA, and RNA sequences distinct from surrounding prose; standard computational tools add deterministic properties as explicit text; and source-derived instruction tasks provide supervision with programmatically checkable answers, such as recovering a masked binding site or locating a genomic feature in JSON. A matched evaluation suite then tests whether those representations transfer. The pipeline's work is to make structured biological knowledge available to a general language model as ordinary text, so that annealing can absorb it wit
Load-bearing premise
The central claim rests on the evaluation being a genuine, contamination-free measure of biological capability; because the evaluation is assembled from the same public benchmark families and source-derived task types used to build the corpus, the reported gains could overstate transfer if the overlap-removal checks missed near-duplicate examples.
What would settle it
Retrain the same base model on TheBioCollection with a stricter and independently implemented decontamination—removing any training record that shares a short sequence segment, entity identifier, or instruction–answer hash with the evaluation—and rerun TheBioCollection-Eval; if the +0.114 corpus effect over text-annealing shrinks materially, the headline gains were memorization rather than transferable biological knowledge.
If this is right
- A model trained with TheBioCollection reaches 0.499 overall on the matched suite versus 0.223 for the base, so biological capability can be added at pre-training scale without changing the architecture.
- The controlled comparison isolates +0.114 to the corpus itself over scientific/web-text annealing, so the gains are not just a side effect of continued pretraining.
- The five general-language benchmarks drop by an average of 0.9 points, suggesting biological and general ability need not trade off sharply.
- Two-hop cross-domain questions improve from 0.313 to 0.507, indicating the corpus supports reasoning across molecule–protein–pathway boundaries, not just single-domain facts.
Where Pith is reading between the lines
- The same recipe of tagged tokens, tool-computed narratives, and verifiable instruction tasks could plausibly be carried to other structured scientific domains, such as materials chemistry or climate data, where heterogeneous databases are also locked in non-text formats.
- Because the evaluation is drawn from the same task families as the training corpus, the +0.114 corpus effect should be re-checked on a pre-registered, independently curated suite to separate transferable knowledge from format memorization.
- The reported weakness in perturbation-response prediction suggests cell-domain records are the current bottleneck; a direct extension would enrich single-cell and perturbation data and re-run the CRISPRi response task.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces TheBioCollection, a 52.6B-token biological pre-training corpus built by consolidating public databases across small molecules, proteins, genomics, cells/pathways, enriching records with tool-computed feature narratives, and adding instruction tasks for protein binding and DNA/RNA localization. It also introduces TheBioCollection-Eval, a matched 1,650-example suite with 18 tasks, and trains the Gravity-16B-A3B base model on the corpus mixed with general scientific/web text. The headline result is that training on TheBioCollection more than doubles the overall biological score (0.223→0.499, Table 3), and a text-annealing-only control is used to attribute +0.114 to the corpus (Table 5), while general-language benchmarks drop by 0.9 points on average.
Significance. If the attribution is sound, this would be a useful contribution: a large, commercially usable biological corpus, a transparent construction pipeline, a matched evaluation suite, and per-task results. The ablation design (same base checkpoint and annealing recipe) is a good idea, and the decontamination details (entity separation, exact/subsequence/15-mer overlap) are reported explicitly. However, the central claim of corpus-specific benefit currently rests on an ablation with unequal token budgets and on an evaluation suite assembled from the same task families used to build the corpus; both need to be addressed before the corpus effect can be considered established.
major comments (2)
- [§3.4 / Table 5] The isolation claim is not yet established. Ours is trained on the 54B-token general text plus the 52.6B-token TheBioCollection, while Text-annealing only is trained on the general text alone; nothing in §3.1 or Table 5 states that the total number of tokens or steps was matched. The +0.114 gap therefore conflates the corpus's content with the effect of ~52.6B additional training tokens. Please report token/step budgets and add a matched-token control (e.g., text-annealing only with the general text replayed so total tokens equal those of Ours).
- [§2.4–2.5 / Tables 2, 11] The evaluation suite is partly circular. TheBioCollection-Eval is assembled from the same public instruction datasets (Mol-Instructions, SMolInstruct, etc.) and source-derived task families (protein binding, DNA/RNA localization) used to construct the corpus. The stated decontamination is entity-level (exact/subsequence/15-mer, sequence hash, coordinates), but it does not remove format/template leakage: the model may memorize the JSON schemas and instruction phrasings rather than acquire transferable biological knowledge. Please report results on independently constructed external benchmarks, or add a format-control ablation (e.g., train on corpus records with answers/targets randomized) to quantify how much of the gain is format familiarity.
minor comments (4)
- [§3.3] The overall score is reported as '0.223 to 0.449', but Table 3 and Table 5 give Ours = 0.499; please correct the typo.
- [Figure 1] 'Homo sapience' should be 'Homo sapiens'.
- [Appendix D] The per-task tables (Tables 7–10) report Base and Ours but not the text-annealing-only condition. Adding that column would let readers verify the nondegeneracy and perturbation-response claims in §3.4.
- [Availability] Please state explicitly where the corpus and evaluation suite will be released and under what licenses, since the paper claims commercial usability and open release.
Circularity Check
No demonstrated circularity; the main evaluation/training overlap is addressed by entity-level decontamination, and the ablation token-count mismatch is a validity concern rather than a circular step.
full rationale
The paper's claimed derivation chain is an empirical training/evaluation loop, not a formal derivation, and no step reduces to its inputs by construction. TheBioCollection-Eval is assembled from benchmarks and source-derived task families that also contribute to the corpus (Sections 2.4 and 2.5, Tables 2 and 11), but the paper explicitly performs entity-level decontamination: 'For any task whose format also appears in training, we separate train and test entities to prevent overlap' and removes exact, subsequence, or shared 15-mer overlap for binder targets. This makes the eval a held-out generalization test within the same task distribution rather than a tautological restatement of the training data; whether the decontamination is sufficient is a leakage/validity question, not circularity. The TA-only vs Ours ablation is not matched in total training tokens: Ours adds the 52.6B-token corpus 'on top of that same general text,' so the +0.114 attribution is confounded by data volume and compute. That is a real correctness risk, but it is not an instance of the enumerated circularity patterns. The only clear self-citation is a minor reference to the authors' earlier VibeProteinBench in a list of scattered resources, and the citation to Trillion Labs for the base model is an artifact reference rather than a load-bearing uniqueness claim. No circular step within the seven enumerated kinds is demonstrated.
Axiom & Free-Parameter Ledger
free parameters (4)
- Heavy-atom contact cutoff for binding interfaces =
5.0 Å
- Eval sampling size per subtask =
100 single-domain; 50 cross-domain
- Training recipe hyperparameters =
seq len 8,192; batch 8.4M tokens; LR 4e-4; weight decay 0.01
- Decontamination thresholds =
exact string; subsequence; shared 15-mer overlap
axioms (3)
- domain assumption Public bio databases and tool outputs (RDKit, DSSP, ESMFold, etc.) are correct enough to serve as training and evaluation labels.
- domain assumption Base Gravity-16B-A3B's pretraining contains no biological corpus, so observed biological gains are attributable to TheBioCollection training.
- domain assumption The decontamination procedure fully prevents train/eval overlap, so gains on TheBioCollection-Eval are not memorization.
read the original abstract
The push toward large language models for biology (BioLM) has created a need for training corpora that can endow models with a genuine understanding of biology. However, existing biological resources, such as molecular databases, protein repositories, genomic annotations, single-cell atlases, and pathway databases, are scattered across heterogeneous formats and remain unorganized into a cohesive corpus for language model training. We present TheBioCollection, a 52.6B-token pre-training-scale corpus that converts these disparate resources into a unified, training-ready form spanning small molecules, proteins, genomic sequences, cells, and pathways. Beyond consolidating existing data, TheBioCollection enriches each record with tool-computed biological properties and introduces new instruction tasks for capabilities that current corpora barely cover. We pair the corpus with TheBioCollection-Eval, a matched suite probing recognition, generation, and prediction across molecular, protein, genomic, cellular, and cross-domain settings. Holding the base Gravity-16B-A3B architecture fixed, training on TheBioCollection more than doubles its overall score on TheBioCollection-Eval with gains in every domain, while leaving general linguistic ability nearly intact.
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2412.07775 , year=
Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets , author=. arXiv preprint arXiv:2412.07775 , year=
-
[2]
Graph diffusion transformers for multi-conditional molecular generation , author=
-
[3]
Nature Biotechnology , volume=
Protein structure prediction with in-cell photo-crosslinking mass spectrometry and deep learning , author=. Nature Biotechnology , volume=
-
[4]
arXiv preprint arXiv:2605.10978 , year=
VibeProteinBench: An Evaluation Benchmark for Language-interfaced Vibe Protein Design , author=. arXiv preprint arXiv:2605.10978 , year=
-
[5]
arXiv preprint arXiv:2502.07527 , year=
Nature language model: deciphering the language of nature for scientific discovery , author=. arXiv preprint arXiv:2502.07527 , year=
-
[6]
arXiv preprint arXiv:2509.21320 , year=
SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines , author=. arXiv preprint arXiv:2509.21320 , year=
-
[7]
arXiv preprint arXiv:2503.06687 , year=
UniGenX: a unified generative foundation model that couples sequence, structure and function to accelerate scientific design across proteins, molecules and materials , author=. arXiv preprint arXiv:2503.06687 , year=
-
[8]
arXiv preprint arXiv:2504.06196 , year=
Txgemma: Efficient and agentic llms for therapeutics , author=. arXiv preprint arXiv:2504.06196 , year=
-
[9]
bioRxiv , year=
BioReason-Pro: Advancing Protein Function Prediction with Multimodal Biological Reasoning , author=. bioRxiv , year=
-
[10]
arXiv preprint arXiv:2306.08018 , year=
Mol-instructions: A large-scale biomolecular instruction dataset for large language models , author=. arXiv preprint arXiv:2306.08018 , year=
-
[11]
MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation , author=
-
[12]
Scaling Atomistic Protein Binder Design with Generative Pretraining and Test-Time Compute , author=
-
[13]
Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations , author=
-
[14]
Findings of the Association for Computational Linguistics , year=
Biot5+: Towards generalized biological understanding with iupac integration and multi-task tuning , author=. Findings of the Association for Computational Linguistics , year=
-
[15]
arXiv preprint arXiv:2406.06316 , year=
Tx-llm: A large language model for therapeutics , author=. arXiv preprint arXiv:2406.06316 , year=
-
[16]
Science , volume=
Evolutionary-scale prediction of atomic-level protein structure with a language model , author=. Science , volume=
-
[17]
Nature Methods , volume=
Nucleotide transformer: building and evaluating robust foundation models for human genomics , author=. Nature Methods , volume=
-
[18]
DNABERT-2: Efficient foundation model and benchmark for multi-species genomes , author=
-
[19]
Nature , volume=
Transfer learning enables predictions in network biology , author=. Nature , volume=
-
[20]
Nature methods , volume=
scGPT: toward building a foundation model for single-cell multi-omics using generative AI , author=. Nature methods , volume=
-
[21]
Science , volume=
Sequence modeling and design from molecular to genome scale with Evo , author=. Science , volume=
-
[22]
Nature , volume=
Genome modelling and design across all domains of life with Evo 2 , author=. Nature , volume=
-
[23]
Evaluating protein transfer learning with TAPE , author=
-
[24]
Peer: a comprehensive and multi-task benchmark for protein sequence understanding , author=
-
[25]
Proteingym: Large-scale benchmarks for protein fitness prediction and design , author=
-
[26]
arXiv preprint arXiv:2311.12570 , year=
Bend: Benchmarking dna language models on biologically meaningful tasks , author=. arXiv preprint arXiv:2311.12570 , year=
-
[27]
arXiv preprint arXiv:2102.09548 , year=
Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development , author=. arXiv preprint arXiv:2102.09548 , year=
-
[28]
Nucleic Acids Research , volume =
PubChem 2025 update , author =. Nucleic Acids Research , volume =
2025
-
[29]
Nucleic Acids Research , volume =
BindingDB in 2024: a FAIR knowledgebase of protein-small molecule binding data , author =. Nucleic Acids Research , volume =
2024
-
[30]
Nucleic Acids Research , volume =
The ChEMBL Database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods , author =. Nucleic Acids Research , volume =
2023
-
[31]
Nucleic Acids Research , volume =
UniProt: the Universal Protein Knowledgebase in 2025 , author =. Nucleic Acids Research , volume =
2025
-
[32]
Nucleic Acids Research , volume =
The Protein Data Bank , author =. Nucleic Acids Research , volume =
-
[33]
Nature , volume =
Expanded encyclopaedias of DNA elements in the human and mouse genomes , author =. Nature , volume =
-
[34]
Nucleic Acids Research , volume =
GENCODE 2021 , author =. Nucleic Acids Research , volume =
2021
-
[35]
Nucleic Acids Research , volume =
RNAcentral 2021: secondary structure integration, improved sequence search and new member databases , author =. Nucleic Acids Research , volume =
2021
-
[36]
Nucleic Acids Research , volume =
Rfam 14: expanded coverage of metagenomic, viral and microRNA families , author =. Nucleic Acids Research , volume =
-
[37]
arXiv preprint arXiv:2402.09391 , year=
Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset , author=. arXiv preprint arXiv:2402.09391 , year=
-
[38]
Cell2Sentence: Teaching Large Language Models the Language of Biology , author =
-
[39]
Science , volume =
The Tabula Sapiens: A multiple-organ, single-cell transcriptomic atlas of humans , author =. Science , volume =
-
[40]
Perturbench: Benchmarking machine learning models for cellular perturbation analysis , author=
-
[41]
Nature Methods , volume =
Human BioMolecular Atlas Program (HuBMAP): 3D Human Reference Atlas construction and usage , author =. Nature Methods , volume =
-
[42]
Nature Methods , volume =
Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations , author =. Nature Methods , volume =
-
[43]
Genetics , volume =
The Gene Ontology knowledgebase in 2023 , author =. Genetics , volume =
2023
-
[44]
Nucleic Acids Research , volume =
The Reactome Pathway Knowledgebase 2024 , author =. Nucleic Acids Research , volume =
2024
-
[45]
eLife , volume =
Systematic integration of biomedical knowledge prioritizes drugs for repurposing , author =. eLife , volume =
-
[46]
2020 , howpublished =
Drug Repurposing Knowledge Graph (DRKG) , author =. 2020 , howpublished =
2020
-
[47]
Nucleic Acids Research , volume =
The STRING database in 2023: protein-protein association networks and functional enrichment analyses for any sequenced genome of interest , author =. Nucleic Acids Research , volume =
2023
-
[48]
Genome Research , volume =
Benchmark and integration of resources for the estimation of human transcription factor activities , author =. Genome Research , volume =
-
[49]
Tissue-based map of the human proteome , author =
Proteomics. Tissue-based map of the human proteome , author =. Science , volume =
-
[50]
2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) , year =
A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding , author =. 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) , year =
2024
-
[51]
bioRxiv , pages=
BioReason-Pro: Advancing Protein Function Prediction with Multimodal Biological Reasoning , author=. bioRxiv , pages=
-
[52]
Biopolymers , volume=
Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features , author=. Biopolymers , volume=
-
[53]
Nucleic Acids Research , volume =
AlphaFold Protein Structure Database 2025: a redesigned interface and updated structural coverage , author =. Nucleic Acids Research , volume =
2025
-
[54]
Nucleic Acids Research , volume =
DrugBank 6.0: the DrugBank Knowledgebase for 2024 , author =. Nucleic Acids Research , volume =
2024
-
[55]
Cell , volume =
A Next Generation Connectivity Map: L1000 Platform and the First 1,000,000 Profiles , author =. Cell , volume =
-
[56]
Nucleic Acids Research , volume =
-
[57]
BioRxiv , year=
PLINDER: The protein-ligand interactions dataset and evaluation resource , author=. BioRxiv , year=
-
[58]
bioRxiv , year=
PINDER: The protein interaction dataset and evaluation resource , author=. bioRxiv , year=
-
[59]
Learning to Design Protein-Protein Interactions with Enhanced Generalization , author =
-
[60]
Morehead, Alex and Chen, Chen and Sedova, Ada and Cheng, Jianlin , journal =
-
[61]
Zhu, Ning and Ming, Yanyu and Zhang, Chengyun and Sen, Cao and Li, Chongyang and Guo, Jingjing and Duan, Hongliang , journal =
-
[62]
Frontiers in Bioinformatics , volume =
Propedia v2.3: A novel representation approach for the peptide-protein interaction database using graph-based structural signatures , author =. Frontiers in Bioinformatics , volume =
-
[63]
Cell , volume =
Mapping information-rich genotype-phenotype landscapes with genome-scale Perturb-seq , author =. Cell , volume =
-
[64]
Science , volume =
Exploring genetic interaction manifolds constructed from rich single-cell phenotypes , author =. Science , volume =
-
[65]
and others , journal =
Ovek Baydar, Damla and Rauluseviciute, Ieva and Aronsen, Dina R. and others , journal =
-
[66]
Kozomara, Ana and Birgaoanu, Maria and Griffiths-Jones, Sam , journal =
-
[67]
and Lowe, Todd M
Chan, Patricia P. and Lowe, Todd M. , journal =
-
[68]
The flan collection: Designing data and methods for effective instruction tuning , author=
-
[69]
bioRxiv , year=
Protenix-v1: Toward high-accuracy open-source biomolecular structure prediction , author=. bioRxiv , year=
-
[70]
doi:10.5281/zenodo.18428170 , url =
Landrum, Greg and Tosco, Paolo and Kelley, Brian and Rodriguez, Ricardo and Cosgrove, David and Vianello, Riccardo and Gedeck, Peter and others , year =. doi:10.5281/zenodo.18428170 , url =
-
[71]
National science review , volume=
Foundation models in bioinformatics , author=. National science review , volume=. 2025 , publisher=
2025
-
[72]
Nature Machine Intelligence , volume=
Generalized biological foundation model with unified nucleic acid and protein language , author=. Nature Machine Intelligence , volume=. 2025 , publisher=
2025
-
[73]
Nature Biotechnology , pages=
Generalist biological artificial intelligence in modeling the language of life , author=. Nature Biotechnology , pages=. 2026 , publisher=
2026
-
[74]
Nature Methods , volume=
Language models for biological research: a primer , author=. Nature Methods , volume=. 2024 , publisher=
2024
-
[75]
Communications Biology , volume=
SaLT&PepPr is an interface-predicting language model for designing peptide-guided protein degraders , author=. Communications Biology , volume=. 2023 , publisher=
2023
-
[76]
Nature methods , volume=
Large-scale foundation model on single-cell transcriptomics , author=. Nature methods , volume=. 2024 , publisher=
2024
-
[77]
Nature Biotechnology , volume=
Predicting transcriptional outcomes of novel multigene perturbations with GEARS , author=. Nature Biotechnology , volume=. 2024 , publisher=
2024
-
[78]
Scientific data , volume=
Building a knowledge graph to enable precision medicine , author=. Scientific data , volume=. 2023 , publisher=
2023
-
[79]
Nature methods , volume=
OmniPath: guidelines and gateway for literature-curated signaling pathway resources , author=. Nature methods , volume=. 2016 , publisher=
2016
-
[80]
Saprot: Protein language modeling with structure-aware vocabulary , author=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.