Pith. sign in

REVIEW 4 major objections 5 minor 92 references

OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniGenBench claims to make genomic foundation model evaluation reproducible by unifying data, models, benchmarks, and interpretability into one modular, one-command platform.

desk verdict Useful benchmarking infrastructure with a real gap between the 'five suites' headline and the delivered BEACON integration; worth a serious referee after the claims are matched to reality. read the letter →

arxiv 2505.14402 v1 pith:BFBY4P75 submitted 2025-05-20 q-bio.GN cs.CL

Heng Yang , Jack Cole , Yuan Li , Renzhi Chen , Geyong Min , Ke Li This is my paper · ORCID
classification q-bio.GNcs.CL
keywords genomicfoundationmodelsbenchmarkingplatformreproducibilityRNAbenchmarksDNAAutoBenchmodelhubinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that research on genomic foundation models suffers a reproducibility crisis caused by fragmented data, model implementations, benchmarks, and interpretability practices, and it presents OmniGenBench as a modular platform that unifies these layers. It claims that with standardized wrappers and automated pipelines, any genomic foundation model can be evaluated with a single command across five benchmark suites containing more than 123 datasets, and that more than 31 open-source models are already integrated. If the platform works as described, model comparisons become reproducible, integration time for new models drops from weeks to days, and interpretability analyses become routine parts of evaluation. The load-bearing assumption is that every bundled suite can be faithfully reproduced through the pipeline; the paper itself reports that BEACON results are not yet reproducible.

What carries the argument

The platform's central object is the standardized wrapper and API layer: model wrappers and tokenizer wrappers that abstract away k-mer, byte-pair, and single-nucleotide tokenization as well as architecture differences, combined with the AutoBench engine that compiles data, model configuration, tasks, and metrics into a single runnable evaluation pipeline. This carries the claim that any model on any genomic task can be benchmarked reproducibly and reduces model integration effort from weeks to about two to three days.

What would settle it

Run the documented one-command AutoBench workflow on a fresh machine for the BEACON suite: if it fails to produce a metrics report, or if the published BEACON dataset remains incomplete as the paper states, then the claim of standardized one-command evaluation across all five suites is not yet true.

Watch

Extended reading notes

Core claim

OmniGenBench is a four-module platform, covering data, model, benchmark, and interpretability layers. It standardizes genomic data into tensors, wraps diverse genomic foundation models behind common APIs regardless of tokenization or architecture, compiles tasks and metrics into an AutoBench engine that runs end-to-end benchmarking with a single command, and ships motif, embedding, and attention tools for interpretability. On top of this, the paper reports a large comparative evaluation of eleven models on four of the five suites, finding that the structure-aware RNA model OmniGenome leads in state-of-the-art counts and average rank, especially on RNA structure tasks, and that its pre-training transfers to plant DNA tasks. It also shows, through sequence-logo analysis, that OmniGenome preserves conserved RNA-family motifs better than other tested models in masked-language-model augmentation.

Load-bearing premise

The five-suite one-command claim rests on the assumption that every external benchmark can be standardized into the automated pipeline without changing task semantics; BEACON currently contradicts this because its results cannot be reproduced and a dataset is missing.

Editorial extensions

If this is right

  • A researcher can evaluate a new genomic foundation model on five suites with a single autobench command rather than reconstructing per-benchmark pipelines.
  • Because more than 31 models share the same interfaces, leaderboard rankings are recomputed automatically and can be audited by rerunning the pipeline.
  • Standardized interpretability tools make motif, embedding, and attention analyses routine parts of evaluation instead of ad-hoc extras.
  • The extensible data hub and model hub let the community add datasets, tasks, metrics, and models under common protocols, so coverage grows without fragmenting.
  • If the five suites run end-to-end, the platform can serve as shared infrastructure for fair, transparent comparison of genomic foundation models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If widely adopted, the platform would let the field treat published model numbers as re-runnable commands, effectively turning leaderboard claims into executable artifacts.
  • The paper's comparative results suggest structure-aware pre-training transfers across RNA and DNA modalities; testing that hypothesis on new structure-aware models would be a natural follow-up.
  • The one-command claim is currently limited in practice because BEACON cannot be reproduced; a fully released and verifiable BEACON integration would be the decisive test of the platform's central promise.
  • A shared 123-dataset registry with versioning could evolve into the de facto standard training and evaluation split for genomics, reducing inconsistencies across papers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents OmniGenBench, a modular benchmarking platform for genomic foundation models (GFMs), with four modules covering data, models, benchmarks, and interpretability. The central claim is that the platform enables standardized, one-command evaluation of any GFM across five benchmark suites (RGB, PGB, GUE, GB, BEACON) with seamless integration of over 31 open-source models. The paper describes the system architecture, provides benchmark results on four suites for eleven GFMs, presents interpretability case studies, and includes appendices with tutorials, task statistics, and a public leaderboard description.

Significance. As an infrastructure contribution, OmniGenBench addresses a real need: GFM benchmarking currently suffers from fragmented codebases, tokenizer incompatibilities, and inconsistent evaluation protocols. The modular design (data, model, benchmark, interpretability), standardized wrappers, tutorials, and the public leaderboard are useful, and the effort to integrate many open-source models is substantial. If the four evaluated suites run as described and the code is released, the platform can lower integration cost for future GFM comparisons. The interpretability case studies (Section 4 and Appendix H) illustrate workflows that are often missing from benchmark papers. However, the central five-suite claim is not yet delivered: BEACON is listed as integrated but has no reproducible results, GUE and GB are evaluated on 10k-sample subsets that are not directly comparable to the original leaderboards, and two RGB tasks are authored by the same group that evaluates its own OmniGenome model. These gaps affect the main contribution claims and must be addressed before the paper can be accepted as a fully reproducible benchmarking infrastructure.

major comments (4)
  1. [Abstract; Section 2.3; Appendix D.7] The abstract and Section 2.3 state that OmniGenBench enables one-command evaluation across five benchmark suites, but Appendix D.7 says that for BEACON 'we are currently unable to provide results because the experimental outcomes cannot be reproduced' and that 'there are still missing dataset.' Section 3 and Appendix D.8 present results for only RGB, PGB, GUE, and GB. The five-suite claim therefore overstates delivered functionality; either BEACON must be made to run end-to-end or the abstract, Figure 1, and Section 2.3 should be revised to state that BEACON is an index-only or in-progress suite.
  2. [Appendix C.3; Appendix C.4; Section 3] The GUE and GB results are computed on randomly stratified 10k-sample subsets, and Appendix C.3 explicitly says they are 'not directly comparable to the original GUE leaderboard.' Section 3, Figure 2, and the SoTA counts nevertheless treat these as full-suite comparisons. The paper should either report the subset status and comparability limits wherever aggregate ranks and SoTA counts appear, or evaluate on the full datasets and provide confidence intervals.
  3. [Table 1; Section 3.1; Appendix D.3] The SNMD and SNMR tasks in RGB are listed with source 'This work,' and the platform includes the authors' own OmniGenome model (Appendix E). The benchmark discussion then uses RGB to argue that OmniGenome's pre-training is superior. Because two of the RGB tasks were constructed by the same group that evaluates its own model, the paper should explicitly separate self-created tasks from independent external benchmarks and should not present aggregate claims such as OmniGenome achieving the best performance across all RGB tasks without this caveat.
  4. [NeurIPS checklist, item 4] The checklist answers 'Yes' to 'Does the paper fully disclose all the information needed to reproduce the main experimental results,' but the paper's own Appendix D.7 states that the BEACON results cannot be reproduced and that a dataset is missing. The checklist justification should be amended to acknowledge the BEACON gap and to state which results are reproducible; as written, the checklist claim is inconsistent with the main text.
minor comments (5)
  1. [Appendix C, first paragraph] The text says 'RGB, PGB, GUE, GB, and BENCON'; 'BENCON' should be 'BEACON.'
  2. [Appendix D.1] The Dropout description says 'we set the Dropout value to 0.1, meaning that no random neuron dropout is applied during training,' which is contradictory; it should say that dropout is applied with probability 0.1.
  3. [Appendix D.2] The entry for OmniGenBench says 'OmniGenBench is the RNA genome FM,' but the model being described is OmniGenome; the platform and model names should be disambiguated throughout.
  4. [Appendix D.7] The phrase 'there are still missing dataset' is ungrammatical, and the passage should specify which dataset(s) are missing because that information is directly relevant to the reproducibility gap.
  5. [Appendix C.6] 'ch-hit-est' and 'pertaining' appear to be typos for 'CD-HIT-EST' and 'pre-training,' respectively.

Circularity Check

1 steps flagged · score 4.0 of 10

Platform functionality is largely independent, but the flagship SoTA claim for OmniGenome rests substantially on a self-authored benchmark (RGB) whose tasks are 'This work' or from the authors' own PlantRNA-FM; the separate 'five benchmark suites' claim is weakened by BEACON's own non-reproducibility note.

  1. self citation load bearing [Section 3.1 (SoTA); Section 2.3 (Benchmark module); Table 1 (Appendix C.1)]
    "Section 3.1: 'OmniGenome notably leads in SoTA achievements, particularly within the RGB and PGB suites.' Section 2.3: 'These include RNA-focused benchmarks (RGB [23], BEACON [14]) and DNA-focused benchmarks (PGB [12], GB [13], GUE [11]).' Table 1 lists 'SNMD ... Source: This work' and 'SNMR ... Source: This work', plus Region-Ara/Rice and TE-Ara/Rice with source [57]."

    The RGB suite is the primary evidence for the paper's SoTA conclusion about OmniGenome, and RGB is cited to the authors' own OmniGenome paper [23]. Six of the eleven RGB tasks in Table 1 come from the current paper itself ('This work') or from PlantRNA-FM [57], whose authors overlap with the present paper. Thus the headline claim 'OmniGenome is best, especially in RGB' is a self-referential evaluation: the benchmark's task definitions and the model being endorsed share the same authorship. The circularity is partial rather than total because the platform's core functionality is independently verifiable on external suites (PGB, GUE, GB) and on RGB's external subtasks, and no fitted parameter is renamed as a prediction.

full rationale

OmniGenBench is primarily a software-engineering contribution: wrapping and standardizing 31 external GFMs, providing data and metric registries, and running external benchmark suites. Those components are self-contained and independently checkable, so the platform itself is not circular. The main self-referential element is confined to the RGB SoTA narrative: RGB is the authors' own benchmark (ref [23]) and includes 'This work' tasks, and the paper uses RGB as leading evidence that OmniGenome is superior; this is a self-evaluation rather than an independent falsification. No equation reduces to its inputs, no fitted parameter is presented as a prediction, and no uniqueness theorem is imported from the authors' prior work. Separately, the abstract's 'five benchmark suites' claim is internally challenged by Appendix D.7, where BEACON results are not provided because 'the experimental outcomes cannot be reproduced' and a dataset is 'still missing'; that is a reproducibility and completeness defect, not a circularity, so it is flagged here but not added to the circularity score.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmarking claims rest on borrowed benchmark tasks, leakage-filtering assumptions, and preprocessing choices. There are no free parameters in the sense of fitted scientific constants; the platform itself is software, not an invented physical entity.

assumptions (3)
  • domain assumption Benchmark tasks and primary metrics from RGB, PGB, GUE, GB, and BEACON are valid and can be reused for GFM evaluation.
    The platform copies benchmark definitions from external sources (Appendix C) and inherits their validity without independent validation.
  • domain assumption Sequence similarity filtering with CD-HIT-EST at 80% and blastn with e-value 1 is sufficient to prevent training/benchmark leakage.
    Appendix C.6 asserts this filtering controls leakage, but the e-value threshold is permissive and no leakage measurement is provided.
  • domain assumption Discarding sequences longer than 512 nucleotides and padding shorter ones preserves the information needed for downstream tasks.
    RGB preprocessing (Appendix C.1) truncates a small fraction of long sequences and pads the rest, which could distort structure or design tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking." pith.science (2026). https://pith.science/paper/BFBY4P75

@misc{pith2026250514402,
  author       = {Pith},
  title        = {Pith review of: OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BFBY4P75}},
  note         = {Machine review of arXiv:2505.14402}
}
read the original abstract

The code of nature, embedded in DNA and RNA genomes since the origin of life, holds immense potential to impact both humans and ecosystems through genome modeling. Genomic Foundation Models (GFMs) have emerged as a transformative approach to decoding the genome. As GFMs scale up and reshape the landscape of AI-driven genomics, the field faces an urgent need for rigorous and reproducible evaluation. We present OmniGenBench, a modular benchmarking platform designed to unify the data, model, benchmarking, and interpretability layers across GFMs. OmniGenBench enables standardized, one-command evaluation of any GFM across five benchmark suites, with seamless integration of over 31 open-source models. Through automated pipelines and community-extensible features, the platform addresses critical reproducibility challenges, including data transparency, model interoperability, benchmark fragmentation, and black-box interpretability. OmniGenBench aims to serve as foundational infrastructure for reproducible genomic AI research, accelerating trustworthy discovery and collaborative innovation in the era of genome-scale modeling.

Figures

Figures reproduced from arXiv: 2505.14402 by the authors.

Figure 1
Figure 1. Overview of the OmniGenBench framework. a) OmniGenBench consists of four core modules covering data, model, benchmark, and interpretability aspects. b) The current release includes 60+ in-silico genomic tasks covering diverse biological processes. c) A four-stage code￾less benchmarking pipeline that can automate the end-to-end evaluation. d) Five benchmark suites (containing 123+ datasets) are indexed in the Data Hu… view at source ↗
Figure 2
Figure 2. State-of-the-Art (SoTA) achievements of public GFMs across tasks within the four primary benchmark suites. We first quantify the SoTA achievements for 11 public GFMs across all evaluated tasks within the aforementioned benchmark suites. A model achieves SoTA by attaining the top performance on a task’s primary metric in our evaluation. This SoTA count intuitively measures cross￾scenario generalization and excellence… view at source ↗
Figure 3
Figure 3. Rank-based radar charts comparing eleven GFMs on the RGB suite. Each small plot [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Sequence logo comparison for RNA families after MLM-based augmentation with different [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Rank-based radar charts comparisons between GFMs across four genomic benchmark 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Overview of GFM parameter scales adapted in [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Screenshot of the interactive web interface for the public leaderboard, illustrating suite [PITH_FULL_IMAGE:figures/full_fig_p032_7.png]
Figure 8
Figure 8. Figure 8: t-SNE visualization of GFM-derived sequence embeddings for antimicrobial resistance [PITH_FULL_IMAGE:figures/full_fig_p039_8.png]
Figure 9
Figure 9. Figure 9: Attention maps (Example 1) for RNA sequences from the bpRNA-1m benchmark. Each [PITH_FULL_IMAGE:figures/full_fig_p041_9.png]
Figure 10
Figure 10. Figure 10: Attention maps (Example 2) for another RNA sequence from the bpRNA-1m dataset. As [PITH_FULL_IMAGE:figures/full_fig_p042_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 68 canonical work pages

  1. [1]

    Molecular structure of nucleic acids: A structure for deoxyri- bose nucleic acid

    James Watson and Francis Crick. Molecular structure of nucleic acids: A structure for deoxyri- bose nucleic acid. Nature, 171:737–738, 1953

    1953

  2. [2]

    Large models for genomics

    Lin Tang. Large models for genomics. Nature Methods, 20:1868, 2023

    2023

  3. [3]

    Why teachers should explore chatgpt’s potential

    Nature Editorial. Why teachers should explore chatgpt’s potential. Nature, 623:15, 2023. doi: 10.1038/d41586-023-03505-5

    work page doi:10.1038/d41586-023-03505-5 2023

  4. [4]

    The opportunities and challenges of chatgpt in education

    Ibrahim Adeshola and Adeola Praise Adepoju. The opportunities and challenges of chatgpt in education. Interactive Learning Environments, 32(10):6159–6172, 2024

    2024

  5. [5]

    Generative artificial intelligence in the metaverse era

    Zhihan Lv. Generative artificial intelligence in the metaverse era. Cognitive Robotics, 3: 208–217, 2023

    2023

  6. [6]

    Beyond reality: The pivotal role of generative ai in the metaverse

    Vinay Chamola, Gaurang Bansal, Tridib Kumar Das, Vikas Hassija, Siva Sai, Jiacheng Wang, Sherali Zeadally, Amir Hussain, Fei Richard Yu, Mohsen Guizani, et al. Beyond reality: The pivotal role of generative ai in the metaverse. IEEE Internet of Things Magazine, 7(4):126–135, 2024

    2024

  7. [7]

    Edelman and Mark Abraham

    David C. Edelman and Mark Abraham. Generative AI will change your business. here’s how to adapt. Harvard Business Review, April 2023. URL https://hbr.org/2023/04/ generative-ai-will-change-your-business-heres-how-to-adapt

    2023

  8. [8]

    A review of chatgpt ai’s impact on several business sectors

    A Shaji George and AS Hovan George. A review of chatgpt ai’s impact on several business sectors. Partners universal international innovation journal, 1(1):9–23, 2023

    2023

Show all 92 references
  1. [9]

    Transformers and genome language models

    Micaela E Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian J Theis, Alan Moses, and Bo Wang. Transformers and genome language models. Nature Machine Intelligence, pages 1–17, 2025

    2025

  2. [10]

    Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome

    Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37(15):2112–2120, 2021

    2021

  3. [11]

    Davuluri, and Han Liu

    Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V . Davuluri, and Han Liu. DNABERT-2: efficient foundation model and benchmark for multi-species genome. CoRR, abs/2306.15006, 2023. doi: 10.48550/ARXIV .2306.15006. URL https://doi.org/10. 48550/arXiv.2306.15006

    2023 doi arXiv

  4. [12]

    A foundational large language model for edible plant genomes

    Javier Mendoza-Revilla, Evan Trop, Liam Gonzalez, Maša Roller, Hugo Dalla-Torre, Bernardo P de Almeida, Guillaume Richard, Jonathan Caton, Nicolas Lopez Carranza, Marcin Skwark, et al. A foundational large language model for edible plant genomes. Communications Biology, 7(1):835, 2024

    2024

  5. [13]

    Genomic benchmarks: a collection of datasets for genomic sequence classification

    Katarína Grešová, Vlastimil Martinek, David ˇCechák, Petr Šimeˇcek, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023

    2023

  6. [14]

    BEACON: benchmark for comprehensive RNA tasks and language models

    Yuchen Ren, Zhiyuan Chen, Lifeng Qiao, Hongtai Jing, Yuchen Cai, Sheng Xu, Peng Ye, Xinzhu Ma, Siqi Sun, Hongliang Yan, Dong Yuan, Wanli Ouyang, and Xihui Liu. BEACON: benchmark for comprehensive RNA tasks and language models. CoRR, abs/2406.10391, 2024. doi: 10.48550/ARXIV .2...

    2024 doi arXiv

  7. [15]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pages 4171–4186. Association for Computational Linguistics, 2019

    2019

  8. [16]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett,...

    2017

  9. [17]

    Evolutionary-scale prediction of atomic-level protein structure with a language model

    Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023

    2023

  10. [18]

    Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y . Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sa...

    2023

  11. [19]

    Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton M

    Eric Nguyen, Michael Poli, Marjan Faizi, Armin W. Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton M. Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Christopher Ré. Hyenadna: Long-range genomic sequence modeling at single nucle...

    2023 doi arXiv

  12. [20]

    Sequence modeling and design from molecular to genome scale with evo

    Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam J Bartie, Armin W Thomas, Samuel H King, Garyk Brixi, et al. Sequence modeling and design from molecular to genome scale with evo. Science, 386(6723):eado9336, 2024

    2024

  13. [21]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752, 2023. doi: 10.48550/ARXIV .2312.00752. URLhttps://doi.org/ 10.48550/arXiv.2312.00752

    2023 doi arXiv

  14. [22]

    Caduceus: Bi-directional equivariant long-range DNA sequence modeling

    Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and V olodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range DNA sequence modeling. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRevi...

    2024

  15. [23]

    Omnigenome: Aligning RNA sequences with secondary structures in genomic foundation models

    Heng Yang and Ke Li. Omnigenome: Aligning RNA sequences with secondary structures in genomic foundation models. CoRR, abs/2407.11242, 2024. doi: 10.48550/ARXIV .2407.11242. URLhttps://doi.org/10.48550/arXiv.2407.11242

    2024 doi arXiv

  16. [24]

    Interpretable rna foundation model from unannotated data for highly accurate rna structure and function predictions

    Jiayang Chen, Zhihang Hu, Siqi Sun, Qingxiong Tan, Yixuan Wang, Qinze Yu, Licheng Zong, Liang Hong, Jin Xiao, Tao Shen, et al. Interpretable rna foundation model from unannotated data for highly accurate rna structure and function predictions. bioRxiv, pages 2022–08, 2022

    2022

  17. [25]

    Turbofold ii: Rna structural alignment and secondary structure prediction informed by multiple homologs

    Zhen Tan, Yinghan Fu, Gaurav Sharma, and David H Mathews. Turbofold ii: Rna structural alignment and secondary structure prediction informed by multiple homologs. Nucleic acids research, 45(20):11570–11581, 2017

    2017

  18. [26]

    bprna: large-scale automated annotation and analysis of rna secondary structure

    Padideh Danaee, Mason Rouches, Michelle Wiley, Dezhong Deng, Liang Huang, and David Hendrix. bprna: large-scale automated annotation and analysis of rna secondary structure. Nucleic acids research, 46(11):5381–5394, 2018

    2018

  19. [27]

    How to benchmark rna secondary structure prediction accuracy

    David H Mathews. How to benchmark rna secondary structure prediction accuracy. Methods, 162:60–67, 2019

    2019

  20. [28]

    Rfam 14: expanded coverage of metagenomic, viral and microrna families

    Ioanna Kalvari, Eric P Nawrocki, Nancy Ontiveros-Palacios, Joanna Argasinska, Kevin Lamkiewicz, Manja Marz, Sam Griffiths-Jones, Claire Toffano-Nioche, Daniel Gautheret, Zasha Weinberg, et al. Rfam 14: expanded coverage of metagenomic, viral and microrna families. Nucleic Acid...

    2021

  21. [29]

    Sars-cov-2 mrna vaccine design enabled by prototype pathogen preparedness

    Kizzmekia S Corbett, Darin K Edwards, Sarah R Leist, Olubukola M Abiona, Seyhan Boyoglu- Barnum, Rebecca A Gillespie, Sunny Himansu, Alexandra Schäfer, Cynthia T Ziwawo, An- thony T DiPiazza, et al. Sars-cov-2 mrna vaccine design enabled by prototype pathogen preparedness. Nat...

    2020

  22. [30]

    Self- supervised learning on millions of pre-mrna sequences improves sequence-based rna splicing prediction

    Ken Chen, Yue Zhou, Maolin Ding, Yu Wang, Zhixiang Ren, and Yuedong Yang. Self- supervised learning on millions of pre-mrna sequences improves sequence-based rna splicing prediction. bioRxiv, pages 2023–01, 2023. 11

    2023

  23. [31]

    Caduceus: Bi-directional equivariant long-range dna sequence modeling

    Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and V olodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024

    2024 arXiv

  24. [32]

    Rna secondary structure prediction using deep learning with thermodynamic integration

    Kengo Sato, Manato Akiyama, and Yasubumi Sakakibara. Rna secondary structure prediction using deep learning with thermodynamic integration. Nature communications, 12(1):941, 2021

    2021

  25. [33]

    Multiple sequence alignment-based rna language model and its application to structural inference

    Yikun Zhang, Mei Lang, Jiuhong Jiang, Zhiqiang Gao, Fan Xu, Thomas Litfin, Ke Chen, Jaswinder Singh, Xiansong Huang, Guoli Song, et al. Multiple sequence alignment-based rna language model and its application to structural inference. Nucleic Acids Research, 52(1):e3–e3, 2024

    2024

  26. [34]

    The kipoi reposi- tory accelerates community exchange and reuse of predictive models for genomics

    Žiga Avsec, Roman Kreuzhuber, Johnny Israeli, Nancy Xu, Jun Cheng, Avanti Shrikumar, Abhimanyu Banerjee, Daniel S Kim, Thorsten Beier, Lara Urban, et al. The kipoi reposi- tory accelerates community exchange and reuse of predictive models for genomics. Nature biotechnology, 37...

    2019

  27. [35]

    Rnabench: A comprehensive library for in silico rna modelling

    Frederic Runge, Karim Farid, Jorg KH Franke, and Frank Hutter. Rnabench: A comprehensive library for in silico rna modelling. bioRxiv, pages 2024–01, 2024

    2024

  28. [36]

    Genbench: A benchmarking suite for systematic evaluation of genomic foundation models

    Zicheng Liu, Jiahui Li, Siyuan Li, Zelin Zang, Cheng Tan, Yufei Huang, Yajing Bai, and Stan Z Li. Genbench: A benchmarking suite for systematic evaluation of genomic foundation models. arXiv preprint arXiv:2406.01627, 2024

    2024 arXiv

  29. [37]

    Diverse genomic embedding benchmark for functional evaluation across the tree of life

    Jacob West-Roberts, Joshua Kravitz, Nishant Jha, Andre Cornman, and Yunha Hwang. Diverse genomic embedding benchmark for functional evaluation across the tree of life. bioRxiv, pages 2024–07, 2024

    2024

  30. [38]

    The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update

    The Galaxy Community. The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update. Nucleic Acids Research, 50(W1):W345–W351, 2022

    2022

  31. [39]

    Proteingym: Large- scale benchmarks for protein fitness prediction and design

    Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: Large- scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing ...

    2024

  32. [40]

    Flip: Benchmark tasks in fitness landscape inference for proteins

    Christian Dallago, Jody Mou, Kadina E Johnston, Bruce J Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pages 2021–11, 2021

    2021

  33. [41]

    Peer: a comprehensive and multi-task benchmark for protein sequence understanding

    Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Ma Chang, Runcheng Liu, and Jian Tang. Peer: a comprehensive and multi-task benchmark for protein sequence understanding. Advances in Neural Information Processing Systems, 35:35156–35173, 2022

    2022

  34. [42]

    Highly accurate protein structure predic- tion with AlphaFold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanis...

    2021

  35. [43]

    Protein complex prediction with alphafold-multimer

    Richard Evans, Michael O’Neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green, Augustin Žídek, Russ Bates, Sam Blackwell, Jason Yim, Olaf Ronneberger, Sebastian Bo- denstein, Michal Zielinski, Alex Bridgland, Anna Potapenko, Andrew Cowie, Kathryn Tunyasu- vuna...

    2021 doi

  36. [44]

    Accurate structure prediction of biomolecular interactions with alphafold 3

    Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1–3, 2024. 12

    2024

  37. [45]

    Language models of protein sequences at the scale of evolution enable accurate structure prediction

    Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022: 500902, 2022

    2022

  38. [46]

    Davuluri

    Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V . Davuluri. DNABERT: pre-trained bidirec- tional encoder representations from transformers model for dna-language in genome.Bioinform., 37(15):2112–2120, 2021

    2021

  39. [47]

    The nucleotide transformer: Building and evaluating robust foundation models for human genomics

    Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. The nucleotide transformer: Building and evaluating robust foundation mo...

    2023

  40. [48]

    Segmentnt: annotating the genome at single-nucleotide resolution with dna foundation models

    Bernardo P de Almeida, Hugo Dalla-Torre, Guillaume Richard, Christopher Blum, Lorenz Hexemer, Maxence Gélard, Javier Mendoza-Revilla, Priyanka Pandey, Stefan Laurent, Marie Lopez, et al. Segmentnt: annotating the genome at single-nucleotide resolution with dna foundation model...

    2024

  41. [49]

    scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data

    Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nat. Mac. Intell. , 4(10):852–866, 2022. doi: 10.1038/S42256-022-00534...

    2022 doi

  42. [50]

    Informative rna base embedding for rna structural alignment and clustering by deep representation learning

    Manato Akiyama and Yasubumi Sakakibara. Informative rna base embedding for rna structural alignment and clustering by deep representation learning. NAR genomics and bioinformatics, 4 (1):lqac012, 2022

    2022

  43. [51]

    Multi-purpose rna language modelling with motif-aware pretraining and type-guided fine-tuning

    Ning Wang, Jiang Bian, Yuchen Li, Xuhong Li, Shahid Mumtaz, Linghe Kong, and Haoyi Xiong. Multi-purpose rna language modelling with motif-aware pretraining and type-guided fine-tuning. Nature Machine Intelligence, pages 1–10, 2024

    2024

  44. [52]

    cdsbert-extending protein language models with codon awareness

    Logan Hallee, Nikolaos Rafailidis, and Jason P Gleghorn. cdsbert-extending protein language models with codon awareness. bioRxiv, 2023

    2023

  45. [53]

    A 5’ utr language model for decoding untranslated regions of mrna and function predictions

    Yanyi Chu, Dan Yu, Yupeng Li, Kaixuan Huang, Yue Shen, Le Cong, Jason Zhang, and Mengdi Wang. A 5’ utr language model for decoding untranslated regions of mrna and function predictions. Nature Machine Intelligence, pages 1–12, 2024

    2024

  46. [54]

    Deciphering 3’utr mediated gene regulation using interpretable deep representation learning

    Yuning Yang, Gen Li, Kuan Pang, Wuxinhao Cao, Xiangtao Li, and Zhaolei Zhang. Deciphering 3’utr mediated gene regulation using interpretable deep representation learning. bioRxiv, pages 2023–09, 2023

    2023

  47. [55]

    Uni-rna: universal pre-trained models revolutionize rna research

    Xi Wang, Ruichu Gu, Zhiyuan Chen, Yongge Li, Xiaohong Ji, Guolin Ke, and Han Wen. Uni-rna: universal pre-trained models revolutionize rna research. bioRxiv, pages 2023–07, 2023

    2023

  48. [56]

    Chatnt: A multimodal conversational agent for dna, rna and protein tasks

    Guillaume Richard, Bernardo P de Almeida, Hugo Dalla-Torre, Christopher Blum, Lorenz Hexemer, Priyanka Pandey, Stefan Laurent, Marie P Lopez, Alexander Laterre, Maren Lang, et al. Chatnt: A multimodal conversational agent for dna, rna and protein tasks. bioRxiv, pages 2024–04, 2024

    2024

  49. [57]

    An interpretable rna foundation model for exploring functional rna motifs in plants

    Haopeng Yu, Heng Yang, Wenqing Sun, Zongyun Yan, Xiaofei Yang, Huakun Zhang, Yiliang Ding, and Ke Li. An interpretable rna foundation model for exploring functional rna motifs in plants. Nature Machine Intelligence, 6(12):1616–1625, 2024

    2024

  50. [58]

    Rna design rules from a massive open laboratory

    Jeehyung Lee, Wipapat Kladwang, Minjae Lee, Daniel Cantu, Martin Azizyan, Hanjoo Kim, Alex Limpaecher, Snehal Gaikwad, Sungroh Yoon, Adrien Treuille, et al. Rna design rules from a massive open laboratory. Proceedings of the National Academy of Sciences , 111(6): 2122–2127, 2014

    2014

  51. [59]

    Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences

    Weizhong Li and Adam Godzik. Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences. Bioinformatics, 22(13):1658–1659, 2006. 13

    2006

  52. [60]

    Basic local alignment search tool

    Stephen F Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David J Lipman. Basic local alignment search tool. Journal of molecular biology, 215(3):403–410, 1990

    1990

  53. [61]

    Viennarna package 2.0

    Ronny Lorenz, Stephan H Bernhart, Christian Höner zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F Stadler, and Ivo L Hofacker. Viennarna package 2.0. Algorithms for molecular biology, 6:1–14, 2011

    2011

  54. [62]

    Carlos Outeiral and Charlotte M. Deane. Codon language embeddings provide strong signals for use in protein engineering. Nature Machine Intelligence, 6:170–179, 2024. doi: 10.1038/ s42256-024-00791-0

    2024

  55. [63]

    Ernie-rna: An rna language model with structure-enhanced representations

    Weijie Yin, Zhaoyu Zhang, Liang He, Rui Jiang, Shuo Zhang, Gan Liu, Xuegong Zhang, Tao Qin, and Zhen Xie. Ernie-rna: An rna language model with structure-enhanced representations. bioRxiv, pages 2024–03, 2024

    2024

  56. [64]

    Mp-rna: Unleashing multi-species rna foundation model via calibrated secondary structure prediction

    Heng Yang and Ke Li. Mp-rna: Unleashing multi-species rna foundation model via calibrated secondary structure prediction. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5278–5296, 2024

    2024

  57. [65]

    Huber, Yue Wan, and Mile Šiki´c

    Rafael Josip Peni´c, Tin Vlaši´c, Roland G. Huber, Yue Wan, and Mile Šiki´c. Rinalmo: General- purpose rna language models can generalize well on structure prediction tasks. arXiv, 2024

    2024

  58. [66]

    Mitigating risks from gene edit- ing and synthetic biology: Global governance priorities

    Carnegie Endowment for International Peace. Mitigating risks from gene edit- ing and synthetic biology: Global governance priorities. Carnegie Endow- ment, 2024. URL https://carnegieendowment.org/research/2024/10/ mitigating-risks-from-gene-editing-and-synthetic-biology-global...

    2024

  59. [67]

    Safety by design: Biosafety and biosecurity in the age of synthetic biology

    iScience. Safety by design: Biosafety and biosecurity in the age of synthetic biology. iScience, 2023. URL https://www.cell.com/iscience/fulltext/S2589-0042%2823% 2900242-0

    2023

  60. [68]

    The economic impact and functional applications of hu- man genetics and genomics, 2021

    American Society of Human Genetics. The economic impact and functional applications of hu- man genetics and genomics, 2021. URL https://www.ashg.org/wp-content/uploads/ 2021/05/ASHG-TEConomy-Impact-Report-Final.pdf

    2021

  61. [69]

    Safety risks and ethical gover- nance of biomedical applications of synthetic biology

    Frontiers in Bioengineering and Biotechnology. Safety risks and ethical gover- nance of biomedical applications of synthetic biology. Frontiers, 2023. URL https://www.frontiersin.org/journals/bioengineering-and-biotechnology/ articles/10.3389/fbioe.2023.1292029/full

    2023

  62. [70]

    Genetic frontiers for conservation: An assessment of synthetic biology and biodiver- sity conservation

    IUCN. Genetic frontiers for conservation: An assessment of synthetic biology and biodiver- sity conservation. 2019. URL https://portals.iucn.org/library/sites/library/ files/documents/2019-012-En.pdf

    2019

  63. [71]

    Ethical framework on risk governance of synthetic biology

    ScienceDirect. Ethical framework on risk governance of synthetic biology. ScienceDirect, 2023. URLhttps://www.sciencedirect.com/science/article/pii/S2588933823000201

    2023

  64. [72]

    Genomic medicine on the frontier of precision medicine

    PMC. Genomic medicine on the frontier of precision medicine. PMC, 2022. URL https: //pmc.ncbi.nlm.nih.gov/articles/PMC9167337/

    2022

  65. [73]

    Innovations in genomics and big data analytics for personalized medicine and health care: A review

    MDPI. Innovations in genomics and big data analytics for personalized medicine and health care: A review. International Journal of Molecular Sciences, 2022. URL https://www.mdpi. com/1422-0067/23/9/4645

    2022

  66. [74]

    Combining ai and crispr will be transformational

    Wired. Combining ai and crispr will be transformational. Wired, 2024. URL https://www. wired.com/story/combining-ai-and-crispr-will-be-transformational

    2024

  67. [75]

    Revolutionizing biological science: The synergy of genomics in agriculture

    Liebert. Revolutionizing biological science: The synergy of genomics in agriculture. Liebert,

  68. [77]

    • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper. • Th...

  69. [78]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discus...

  70. [79]

    • All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 45 Justification: Guidelines: • The answer NA means that the paper does not include theoretical results. • A...

  71. [80]

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  72. [81]

    • Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 46 Answer: [Yes] Justification: Guidelines: • The answer...

  73. [82]

    • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Guidelines: • The answer NA m...

  74. [83]

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: The evaluation scale is tremendous for repreating i...

  75. [84]

    • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Guidelines: • The answe...

  76. [85]

    • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethicshttps://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS...

  77. [86]

    • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Guidelines: • The answer NA means that there is no societal impact of the work performed. • If the author...

  78. [87]

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Guidelines: • T...

  79. [88]

    • The authors should cite the original paper that produced the code package or dataset

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Guidelines: • Th...

  80. [89]

    • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 49 Answer: [Yes] Justification: Guidelines: • The answer NA means that the paper does not release new assets. • Researchers should communicate th...

  81. [90]

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  82. [91]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  83. [92]

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  84. [2023]

    Cls” = classification, “Reg

    URL https://www.liebertpub.com/doi/10.1089/omi.2023.0197. 14 Table of Contents in Appendices A Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.1 Benchmarking Platforms an...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.