Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Omni-DNA: A Unified Genomic Foundation Model for Cross-Modal and Multi-Task Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Omni-DNA argues that a single family of autoregressive transformers, pretrained on DNA alone and finetuned once on many tasks, can match or beat task-specific genomic models and generate functional text and images from raw sequence.

desk verdict The benchmark story is plausible and useful, but the DNA2Func cross-modal claim is overclaimed: the evaluation only scores RNA type, not function. read the letter →

arxiv 2502.03499 v1 pith:R37X2X5W submitted 2025-02-05 q-bio.GN cs.AIcs.LG

Zehui Li , Vallijah Subasri , Yifei Shen , Dongsheng Li , Yiren Zhao , Guy-Bart Stan , Caihua Shan This is my paper · ORCID
classification q-bio.GNcs.AIcs.LG
keywords genomicfoundationmodelnexttokenpredictionmulti-taskfinetuningcross-modallearningDNA-to-textDNA-to-imagevocabularyexpansionhistonemodification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Omni-DNA sets out to show that the recipe behind large language models—next-token pretraining on a large corpus, then finetuning—can make a single genomic foundation model handle many tasks and output modalities at once, instead of requiring a separate finetuned copy per task. The authors pretrain autoregressive transformers from 20 million to 1 billion parameters on roughly 300 billion nucleotides, then expand the tokenizer with task-specific tokens and finetune on unified sequence-to-token tasks. They report state-of-the-art results on 18 of 26 tasks in the Nucleotide Transformer and Genomic Benchmark suites, and show one finetuned model solving ten histone acetylation and methylation tasks at once, outperforming separate single-task models. They also introduce DNA2Func and Needle-in-DNA, in which the same model maps DNA to free-form functional text and to handwritten-digit images, arguing that a generalist genomic model can generate labels, language, and images from sequence alone. If right, this removes the storage and latency overhead of K separate finetuned models and opens genomic annotation to generative outputs.

What carries the argument

The load-bearing mechanism is a two-stage recipe: next-token pretraining on DNA, then cross-modal multi-task finetuning. During finetuning the tokenizer's vocabulary is expanded with task-specific prompt and response tokens; a distribution-shift calculation (the paper's Eq. 3) shows that adding new tokens reduces every existing token's probability, and the paper counters this with Important Key Repetition, which replicates label tokens by a factor α, and with NEFTune noisy embeddings. A task-unification step rewrites every task as sequence + instruction → response tokens, and a VQ-VAE discretizes continuous outputs such as images into tokens so that DNA2Image becomes next-token generation. Architectural choices—RoPE over ALiBi, and a bias-free non-parametric LayerNorm on the two largest models—are part of what makes the pretraining stable enough for this unified finetuning to work.

What would settle it

Replace the LLM-written descriptions in Seq2Func with descriptions shuffled across sequences while keeping the same labels, then finetune and re-evaluate with the same LLM judge; if F1 and MCC stay near the reported values, the DNA2Func result reflects imitation of annotation style rather than sequence–function mapping.

Watch

Extended reading notes

Core claim

Omni-DNA is a family of autoregressive transformers (20M to 1B parameters) pretrained with next-token prediction on roughly 300 billion nucleotides drawn from a deduplicated multi-species DNA corpus. The paper's central claim is that after pretraining, expanding the tokenizer with task-specific prompt and response tokens and finetuning on several tasks at once produces a generalist genomic model: on the Nucleotide Transformer and Genomic Benchmark suites it reports state-of-the-art results on 18 of 26 tasks, with the 1B model reaching an average of 0.767 across the 18 NT tasks and the 116M model reaching 0.879 across the eight GB tasks. Finetuned jointly on ten acetylation and methylation tasks, the same 1B model reaches 0.739 average accuracy, above any of the separately finetuned single-task models. The paper further introduces two cross-modal tasks—DNA2Func, mapping sequences to natural-language functional descriptions, and Needle-in-DNA, mapping sequences to handwritten-digit images—and reports that Omni-DNA generates fluent function text and near-perfect digit images (macro F1 0.987) from sequence input, indicating that a single DNA-pretrained transformer can emit labels, text, and images.

Load-bearing premise

The cross-modal DNA2Func claim assumes the LLM-generated functional descriptions in Seq2Func actually reflect biology; if those texts only mirror the annotator's style, the DNA-to-text result may not demonstrate true functional understanding.

Editorial extensions

If this is right

  • A single Omni-DNA model finetuned once on the ten histone acetylation and methylation tasks beats ten separately finetuned models, so the cost of finetuning and storing K task-specific models collapses to one model.
  • Multi-task finetuning on biologically related tasks can improve average accuracy instead of sacrificing it, a synergistic effect that makes grouping related genomic tasks a practical strategy.
  • Any continuous high-dimensional genomic output can be converted into a token-generation task through VQ-VAE discretization, so the same autoregressive model can in principle emit assay predictions or structures.
  • DNA-to-text generation gives genomic models a free-form output channel, so functional annotation can become generative rather than limited to fixed label sets.
  • Autoregressive next-token pretraining can match or beat bidirectional masked-language-model pretraining on standard genomic classification benchmarks, making the NTP paradigm a credible default for future genomic foundation models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: the same vocabulary-expansion and VQ-VAE recipe could be pointed at other continuous genomic outputs—RNA-seq coverage, chromatin profiles, or 3D structures—if a codebook can discretize them faithfully, making one model a general genomic encoder–decoder.
  • The LLM-generated Seq2Func labels mean the DNA2Func result is only as strong as those synthetic texts' biological fidelity; a validation against experimentally curated functional annotations would separate sequence-based understanding from annotation-style imitation.
  • The near-perfect Needle-in-DNA score suggests synthetic motif-to-image tasks can act as controlled probes for whether a pretrained model encodes specific motif patterns, an idea that could extend to de novo motif discovery by testing which sequences trigger which generated images.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Omni-DNA, a family of autoregressive transformer-based genomic foundation models (20M to 1B parameters) pretrained with next-token prediction on roughly 300 billion deduplicated nucleotides, followed by vocabulary expansion and multi-task finetuning. The authors report state-of-the-art results on 18 of 26 tasks across the Nucleotide Transformer (NT) and Genomic Benchmark (GB) suites, present a multi-task finetuning experiment on ten histone-related tasks, and introduce two cross-modal tasks: DNA2Func, which maps DNA sequences to textual functional descriptions, and Needle-in-DNA, which maps DNA sequences containing known motifs to handwritten-digit images. The stated goal is a generalist genomic model that can handle classification, text, and image outputs with a single finetuning procedure.

Significance. If the NT/GB benchmark results hold, the paper makes a useful empirical contribution by showing that a carefully configured next-token-prediction autoregressive model can match or exceed bidirectional MLM-based models on standard regulatory classification tasks, and the released model family is a practical resource. The multi-task finetuning result on ten histone tasks is interesting even though the gains are average-based rather than per-task. The cross-modal DNA2Func claim, however, is currently supported by an evaluation loop in which an LLM generates the functional text and a different LLM judges the output by assigning it to one of seven coarse RNA types; as presented, this does not demonstrate functional or gene-level understanding. The conventional benchmark contribution is therefore the strongest part of the paper, while the cross-modal generalization claim needs substantially more evidence or a more modest framing.

major comments (3)
  1. [§5.2, Appendix F, Table 12] The DNA2Func evaluation does not support the abstract's claim of mapping DNA sequences to textual functional descriptions. The Seq2Func dataset is constructed by taking RefSeq sequences already labeled with seven RNA types, asking an LLM annotator to expand those labels into fluent text, and then scoring generated text with a GPT-4o judge that classifies it back into the same seven labels. Table 12 makes the gap concrete: for an input whose ground truth is 'mRNA:CD209/DC-SIGN', Omni-DNA@ft produces a fluent description of CCL22; because CCL22 is also an mRNA, the output is counted as correct at the type level even though the specific function is wrong. A model that emits class-conditioned fluent text on the basis of coarse sequence features could reproduce the reported F1/MCC without knowing which gene or biological process is present. I therefore do not regard the cross-modal functional-understanding claim as established. Please add a gene-level or entity-level evaluation (e.g., exact gene-name matching, human assessment of biological correctness), report the reliability and variance of the LLM judge, or explicitly reframe the task as seven-way type-conditional text generation rather than functional description generation.
  2. [Abstract; §5.1, Table 4] The claim that multi-task finetuning 'surpasses models trained on each task individually' is stronger than the data show. In Table 4, Omni-DNA@mult. achieves the highest average accuracy (0.739 vs. 0.694 for Omni-DNA@sgl.) but wins on only 4 of 10 tasks and is worse than the single-task model on H3, H3K36ME3, H3K4ME1, H3K79ME3, H3K9AC, and H4. The abstract and Section 5.1 should be rephrased to say that multi-task finetuning improves average accuracy and can be synergistic on some tasks, with the per-task trade-offs discussed explicitly.
  3. [§4.1, §4.2, Appendix D, Tables 8-9] The headline 'state-of-the-art on 18 out of 26 tasks' should be qualified. The comparison mixes numbers reported by other papers with numbers produced by this paper's own finetuning pipeline; for the three splice-site tasks Omni-DNA is below NT models, and the 18/26 count combines 13/18 on NT and 5/8 on GB using different sizes of the Omni-DNA family. Please state explicitly which baseline numbers were produced with the identical pipeline in this paper, report the 18/26 count as a comparison against published baselines under the Caduceus evaluation setup rather than as an unconditional SOTA claim, and give a per-model-size breakdown so readers can see which Omni-DNA size attains each SOTA.
minor comments (6)
  1. [§3.2] The text cites 'NEFTune (Press et al., 2021)'; NEFTune is Jain et al. (2023), while Press et al. is the ALiBi reference. Please correct the citation.
  2. [§5.2 vs. Appendix F] Section 5.2 states the Seq2Func dataset contains 330,000 sequences, while Appendix F's final composition says 300,000 DNA-function pairs; the counts should be reconciled.
  3. [Table 14] The ground-truth label 'nRNA:U6 spliceosomal RNA' should read 'snRNA'.
  4. [Table 9] Several entries use '+' where the intended symbol is '±' (e.g., 0.525 + 0.019, 0.407 + 0.041, 0.795 + 0.091, 0.945 + 0.016, 0.927 + 0.023); please fix the typesetting.
  5. [§5.3] The Needle-in-DNA human evaluation reports two annotators but does not report inter-annotator agreement or how many samples were discarded as inconsistent; please provide these statistics.
  6. [Conclusion] The conclusion contains the typo 'DNA2Funtion'; it should be 'DNA2Function'.
Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper relies on standard ML hyperparameters (learning rate, batch size) that are not counted here, but the task-tuned parameters (alpha, noise level, VQ-VAE codebook size) are free parameters because they are chosen based on the target tasks. The main axioms are domain assumptions about tokenization, pretraining data sufficiency, and the validity of LLM-generated annotations as ground truth. No invented entities are introduced.

free parameters (4)
  • NEFTune noise level beta = 5
    Chosen in Section 5.1 for multi-task finetuning; affects all multi-task results and is tuned on the target tasks.
  • Key token replication factor alpha = 10
    Selected via ablation in Section 6.3; central to making classification work in the unified token format.
  • VQ-VAE codebook size K = 6
    Set in Section 5.3 for Needle-in-DNA; determines the discrete token space for image generation.
  • VQ-VAE compression ratio r = 4
    Fixed in Section 5.3; reduces 28x28 images to 49 tokens and affects reconstruction fidelity.
assumptions (5)
  • domain assumption BPE tokenization on DNA with 4096 vocab and vocabulary expansion does not destroy functional signal.
    Section 3.1 (iii) and Section 3.2 rely on BPE tokens being a valid representation for both pretraining and finetuning.
  • domain assumption The deduplicated NCBI multi-species genome dataset is a sufficient pretraining corpus for learning generalizable genomic representations.
    Section 3.1 states pretraining on this corpus; if the corpus is unrepresentative, downstream transfer would fail.
  • domain assumption Autoregressive next-token prediction on DNA captures genomic syntax relevant to downstream tasks.
    The whole approach in Section 3.1 is built on this premise; it is the stated reason for choosing NTP over MLM.
  • ad hoc to paper LLM-generated functional annotations in Seq2Func are acceptable ground truth for DNA function.
    Appendix F constructs the dataset by prompting an LLM annotator to expand short annotations; the resulting text is treated as ground truth.
  • domain assumption The VQ-VAE discretization preserves task-relevant information for image generation.
    Section 3.2 and Section 5.3 assume the learned discrete codes retain enough information to reconstruct and classify images.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Omni-DNA: A Unified Genomic Foundation Model for Cross-Modal and Multi-Task Learning." pith.science (2026). https://pith.science/paper/R37X2X5W

@misc{pith2026250203499,
  author       = {Pith},
  title        = {Pith review of: Omni-DNA: A Unified Genomic Foundation Model for Cross-Modal and Multi-Task Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R37X2X5W}},
  note         = {Machine review of arXiv:2502.03499}
}
read the original abstract

Large Language Models (LLMs) demonstrate remarkable generalizability across diverse tasks, yet genomic foundation models (GFMs) still require separate finetuning for each downstream application, creating significant overhead as model sizes grow. Moreover, existing GFMs are constrained by rigid output formats, limiting their applicability to various genomic tasks. In this work, we revisit the transformer-based auto-regressive models and introduce Omni-DNA, a family of cross-modal multi-task models ranging from 20 million to 1 billion parameters. Our approach consists of two stages: (i) pretraining on DNA sequences with next token prediction objective, and (ii) expanding the multi-modal task-specific tokens and finetuning for multiple downstream tasks simultaneously. When evaluated on the Nucleotide Transformer and GB benchmarks, Omni-DNA achieves state-of-the-art performance on 18 out of 26 tasks. Through multi-task finetuning, Omni-DNA addresses 10 acetylation and methylation tasks at once, surpassing models trained on each task individually. Finally, we design two complex genomic tasks, DNA2Function and Needle-in-DNA, which map DNA sequences to textual functional descriptions and images, respectively, indicating Omni-DNA's cross-modal capabilities to broaden the scope of genomic applications. All the models are available through https://huggingface.co/collections/zehui127

Figures

Figures reproduced from arXiv: 2502.03499 by the authors.

Figure 1
Figure 1. Demonstration of Omni-DNA’s cross-modal capabili￾ties. Given a DNA sequence, Omni-DNA could generate a natural language description for functional annotations. Genomic Foundation Models (GFMs) (Nguyen et al., 2024; Zhou et al., 2023; Dalla-Torre et al., 2024; Schiff et al., 2024), a type of Deep Neural Networks (DNNs) for ge￾nomic sequence modeling, has emerged as essential tools to automate the annotation process. … view at source ↗
Figure 2
Figure 2. Accuracy comparison of Omni-DNA@mult. against Omni-DNA@sgl. and baselines across 10 NT tasks. Omni￾DNA@mult. achieves the highest average accuracy. (3) We evaluate Omni-DNA on novel cross-modal and multi￾task genomic tasks, such as DNA2Func and DNA2Image. 2. Preliminaries and Notations 2.1. Genomic Sequence Modeling DNA is a polymer made of up four types of nucleotides: Adenine (A), Thymine (T), Guanine (G), and Cyt… view at source ↗
Figure 3
Figure 3. (a) Cross Entropy Loss on test set during pretraining. The models with varying sizes show a stable decrease in loss. (b) No-Bias Normalization Layer stabilizes the average value of feed-forward weights in transformer layers. This pattern is consistent across all the transformer blocks. using cross-entropy loss on the target tokens: pθ(y (i) |x (i) ) = XN i=1 TX′ t=1 log pθ(y (i) t |y (i) 1:t−1 , x (i) ). (1) Notably… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Overview of of Omni-DNA architecture. In pretraining, Omni-DNA are trained on DNA only with next-token prediction. Multi-task finetuning enables the model to perform diverse tasks including classification, function prediction, and DNA-to-image [PITH_FULL_IMAGE:figures…
Figure 5
Figure 5. Figure 5: F1 scores and invalid percentages for Needle-in-DNA, averaged and per class. Omni-DNA outperforms both baselines. , }, sampled from the MNIST dataset (Yadav & Bottou, 2019). The dataset is split into finetuning, validation, and test sets in an 8:1:1 ratio. Following da…
Figure 6
Figure 6. Figure 6: Losses for Omni-DNA (116M) pretrained with ALiBi and RoPE, indicating RoPE’s faster convergence and lower losses. degradation varies with model size in the promoter TATA classification task. Notably, increasing the model size allevi￾ates this issue, and Omni-DNA (1B) m…
Figure 7
Figure 7. Figure 7: Impact of model size on performance degradation due to vocabulary expansion. Larger models exhibit better resilience against distribution shifts. 6.3. Impact of Token Replication Factor [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: illustrates the effect of the token replication factor α on the promoter TATA classification task for Omni-DNA sized 116M and 1B. Without token replication (α = 1), classification performance is poor. We find that setting α within the range [8, 11] is effective across …
Figure 9
Figure 9. Figure 9: Images Generated by Omni-DNA. These digits are novel in the sense that they do not exist in the training set. F. Functional Annotation Generation Input: ATGGACG... Prompt: describe functions of given Short Function gene/rna precisely and concisely Annotation Lnc_RNA 27…
Figure 10
Figure 10. Figure 10: Seq2Func Dataset Construction and Evaluation Pipeline. Genomic sequences from 20 species are annotated with functional descriptions using an LLM, generating 300,000 DNA-function pairs. The evaluation pipeline compares model predictions against ground truth, with an LL…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Frozen but Not Always Accessible: A Representation Analysis of Genomic Language Models

    q-bio.GN 2026-08 conditional novelty 5.0 of 10

    Frozen genomic language model embeddings recover most fine-tuned performance on promoter tasks (95-100%) but only 60-88% on splice-site tasks, a gap that persists across readouts and model families.

Reference graph

Works this paper leans on

54 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    J., Bambrick, J., et al

    Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp.\ 1--3, 2024

    2024

  2. [2]

    T., and Frey, B

    Alipanahi, B., Delong, A., Weirauch, M. T., and Frey, B. J. Predicting the sequence specificities of dna-and rna-binding proteins by deep learning. Nature Biotechnology, 33 0 (8): 0 831--838, 2015

    work page 2015

  3. [3]

    E., Phalke, N., , Z

    Andrews, G., Fan, K., Pratt, H. E., Phalke, N., , Z. C., Karlsson, E. K., Lindblad-Toh, K., Gazal, S., Moore, J. E., Weng, Z., et al. Mammalian evolution of human cis-regulatory elements and transcription factor binding sites. Science, 380 0 (6643): 0 eabn7930, 2023

    work page 2023

  4. [4]

    R., Grabska-Barwinska, A., Taylor, K

    Avsec, Z ., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska-Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P., and Kelley, D. R. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 0 (10): 0 1196--1203, 2021 a

    work page 2021

  5. [5]

    Base-resolution models of transcription-factor binding reveal soft motif syntax

    Avsec, Z ., Weilert, M., Shrikumar, A., Krueger, S., Alexandari, A., Dalal, K., Fropf, R., McAnany, C., Gagneur, J., Kundaje, A., et al. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nature genetics, 53 0 (3): 0 354--366, 2021 b

    work page 2021

  6. [6]

    Ba, J. L. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  7. [7]

    S., and Song, Y

    Benegas, G., Batra, S. S., and Song, Y. S. Dna language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120 0 (44): 0 e2311219120, 2023

    2023

  8. [8]

    H., Zielinski, M., Sargeant, T., et al

    Cheng, J., Novati, G., Pan, J., Bycroft, C., Z emgulyt \.e , A., Applebaum, T., Pritzel, A., Wong, L. H., Zielinski, M., Sargeant, T., et al. Accurate proteome-wide missense variant effect prediction with alphamissense. Science, 381 0 (6664): 0 eadg7492, 2023

    work page 2023

Show all 54 references
  1. [9]

    Dnalongbench: A benchmark suite for long-range dna prediction tasks

    Cheng, W., Song, Z., Zhang, Y., Wang, S., Wang, D., Yang, M., Li, L., and Ma, J. Dnalongbench: A benchmark suite for long-range dna prediction tasks. bioRxiv, pp.\ 2025--01, 2025

    2025

  2. [10]

    H., Oteri, F., Dallago, C., Trop, E., de Almeida, B

    Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Method...

    2024

  3. [11]

    P., Dalla-Torre, H., Richard, G., Blum, C., Hexemer, L., G \'e lard, M., Mendoza-Revilla, J., Pandey, P., Laurent, S., Lopez, M., et al

    de Almeida, B. P., Dalla-Torre, H., Richard, G., Blum, C., Hexemer, L., G \'e lard, M., Mendoza-Revilla, J., Pandey, P., Laurent, S., Lopez, M., et al. Segmentnt: annotating the genome at single-nucleotide resolution with dna foundation models. bioRxiv, pp.\ 2024--03, 2024

    2024

  4. [12]

    Genomic benchmarks: a collection of datasets for genomic sequence classification

    Gre s ov \'a , K., Martinek, V., C ech \'a k, D., S ime c ek, P., and Alexiou, P. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24 0 (1): 0 25, 2023

    2023

  5. [13]

    H., Ivison, H., Magnusson, I., Wang, Y., et al

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  6. [14]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  7. [15]

    Initializing new word embeddings for pretrained language models

    Hewitt, J. Initializing new word embeddings for pretrained language models. URL: https:/nlp. stanford. edu/\ johnhew//vocab-expansion. html , 2021

    2021

  8. [16]

    R., Kailkhura, B., Schwarzschild, A., Saha, A., et al

    Jain, N., Chiang, P.-y., Wen, Y., Kirchenbauer, J., Chu, H.-M., Somepalli, G., Bartoldson, B. R., Kailkhura, B., Schwarzschild, A., Saha, A., et al. Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint arXiv:2310.05914, 2023

    arXiv

  9. [17]

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  10. [18]

    Genomic data resources: challenges and promises

    Lathe, W., Williams, J., Mangan, M., and Karolchik, D. Genomic data resources: challenges and promises. Nature Education, 1 0 (3): 0 2, 2008

    2008

  11. [19]

    Li, S., Wang, Z., Liu, Z., Wu, D., Tan, C., Zheng, J., Huang, Y., and Li, S. Z. Vqdna: Unleashing the power of vector quantization for multi-species genomic sequence modeling. arXiv preprint arXiv:2405.10812, 2024 a

    2024 arXiv

  12. [20]

    Gv-rep: A large-scale dataset for genetic variant representation learning

    Li, Z., Subasri, V., Stan, G.-B., Zhao, Y., and Wang, B. Gv-rep: A large-scale dataset for genetic variant representation learning. arXiv preprint arXiv:2407.16940, 2024 b

    2024 arXiv

  13. [21]

    Linder, J., Srivastava, D., Yuan, H., Agarwal, V., and Kelley, D. R. Predicting rna-seq coverage from dna sequence as a unifying model of gene regulation. Nature Genetics, pp.\ 1--13, 2025

    2025

  14. [22]

    Reframing instructional prompts to gptk's language

    Mishra, S., Khashabi, D., Baral, C., Choi, Y., and Hajishirzi, H. Reframing instructional prompts to gptk's language. arXiv preprint arXiv:2109.07830, 2021

    2021 arXiv

  15. [23]

    Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution

    Nguyen, E., Poli, M., Faizi, M., Thomas, A., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024

    2024

  16. [24]

    Oubounyt, M., Louadi, Z., Tayara, H., and Chong, K. T. Deepromoter: robust promoter predictor using deep learning. Frontiers in genetics, 10: 0 286, 2019

    2019

  17. [25]

    Byte latent transformer: Patches scale better than tokens

    Pagnoni, A., Pasunuru, R., Rodriguez, P., Nguyen, J., Muller, B., Li, M., Zhou, C., Yu, L., Weston, J., Zettlemoyer, L., et al. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09871, 2024

    2024 arXiv

  18. [26]

    K., Harbison, C

    Pokholok, D. K., Harbison, C. T., Levine, S., Cole, M., Hannett, N. M., Lee, T. I., Bell, G. W., Walker, K., Rolfe, P. A., Herbolsheimer, E., et al. Genome-wide map of nucleosome acetylation and methylation in yeast. Cell, 122 0 (4): 0 517--527, 2005

    2005

  19. [27]

    A., and Lewis, M

    Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021

    2021 arXiv

  20. [28]

    Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna

    Qiao, L., Ye, P., Ren, Y., Bai, W., Liang, C., Ma, X., Dong, N., and Ouyang, W. Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna. arXiv preprint arXiv:2412.13716, 2024

    2024 arXiv

  21. [29]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  22. [30]

    W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al

    Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  23. [31]

    M., and Poetsch, A

    Sanabria, M., Hirsch, J., Joubert, P. M., and Poetsch, A. R. Dna language model grover learns sequence context in the human genome. Nature Machine Intelligence, 6 0 (8): 0 911--923, 2024

    2024

  24. [32]

    H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T

    Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021

    arXiv

  25. [33]

    Caduceus: Bi-directional equivariant long-range dna sequence modeling

    Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, V. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024

    2024 arXiv

  26. [34]

    Bio-xlstm: Generative modeling, representation and in-context learning of biological and chemical sequences

    Schmidinger, N., Schneckenreiter, L., Seidl, P., Schimunek, J., Hoedt, P.-J., Brandstetter, J., Mayr, A., Luukkonen, S., Hochreiter, S., and Klambauer, G. Bio-xlstm: Generative modeling, representation and in-context learning of biological and chemical sequences. arXiv preprin...

    2024 arXiv

  27. [35]

    L., Ciufo, S., Domrachev, M., Hotton, C

    Schoch, C. L., Ciufo, S., Domrachev, M., Hotton, C. L., Kannan, S., Khovanskaya, R., Leipe, D., Mcveigh, R., O’Neill, K., Robbertse, B., et al. Ncbi taxonomy: a comprehensive update on curation, resources and tools. Database, 2020: 0 baaa062, 2020

    2020

  28. [36]

    Neural machine translation of rare words with subword units

    Sennrich, R. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  29. [37]

    Multi-task inference: Can large language models follow multiple instructions at once? arXiv preprint arXiv:2402.11597, 2024

    Son, G., Baek, S., Nam, S., Jeong, I., and Kim, S. Multi-task inference: Can large language models follow multiple instructions at once? arXiv preprint arXiv:2402.11597, 2024

    2024 arXiv

  30. [38]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  31. [39]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  32. [40]

    L., Church, G

    Tompa, M., Li, N., Bailey, T. L., Church, G. M., De Moor, B., Eskin, E., Favorov, A. V., Frith, M. C., Fu, Y., Kent, W. J., et al. Assessing computational tools for the discovery of transcription factor binding sites. Nature Biotechnology, 23 0 (1): 0 137--144, 2005

    2005

  33. [41]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  34. [42]

    Neural discrete representation learning

    Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  35. [43]

    J., and Davey, N

    Van Roey, K., Gibson, T. J., and Davey, N. E. Motif switches: decision-making in cell regulation. Current opinion in structural biology, 22 0 (3): 0 378--385, 2012

    2012

  36. [44]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  37. [45]

    Multi-purpose rna language modelling with motif-aware pretraining and type-guided fine-tuning

    Wang, N., Bian, J., Li, Y., Li, X., Mumtaz, S., Kong, L., and Xiong, H. Multi-purpose rna language modelling with motif-aware pretraining and type-guided fine-tuning. Nature Machine Intelligence, pp.\ 1--10, 2024

    2024

  38. [46]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  39. [47]

    Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data

    Xin, H., Guo, D., Shao, Z., Ren, Z., Zhu, Q., Liu, B., Ruan, C., Li, W., and Liang, X. Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data. arXiv preprint arXiv:2405.14333, 2024

    2024 arXiv

  40. [48]

    and Bottou, L

    Yadav, C. and Bottou, L. Cold case: The lost mnist digits. Advances in neural information processing systems, 32, 2019

    2019

  41. [49]

    C., Sribour, M., Baek, I., Laurin, M., and Fuchs, E

    Yang, Y., Gomez, N., Infarinato, N., Adam, R. C., Sribour, M., Baek, I., Laurin, M., and Fuchs, E. The pioneer factor sox9 competes for epigenetic factors to switch stem cell fates. Nature Cell Biology, 25 0 (8): 0 1185--1195, 2023

    2023

  42. [50]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  43. [51]

    Towards lifelong learning of large language models: A survey

    Zheng, J., Qiu, S., Shi, C., and Ma, Q. Towards lifelong learning of large language models: A survey. arXiv preprint arXiv:2406.06391, 2024

    2024 arXiv

  44. [52]

    and Troyanskaya, O

    Zhou, J. and Troyanskaya, O. G. Predicting effects of noncoding variants with deep learning--based sequence model. Nature methods, 12 0 (10): 0 931--934, 2015

    2015

  45. [53]

    Dnabert-2: Efficient foundation model and benchmark for multi-species genome

    Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R., and Liu, H. Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023

    2023 arXiv

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.