REVIEW 4 major objections 6 minor 24 references
Multi-modal single-cell foundation models via dynamic token adaptation
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Fusing DNA sequence and single-cell transcriptomes lets a model predict which genes a mutation hits.
desk verdict A genuinely new adapter idea with an evaluation that doesn't yet prove the headline claim; the missing negative-control perturbation is the main fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is dynamic token adaptation: instead of a fixed, learnable embedding per gene token (as in standard BERT or Geneformer), each gene's input embedding is computed on the fly by mean-pooling Enformer's 3,072-dimensional DNA-sequence embeddings over a 196,608 bp window around the gene's transcriptional start site and passing them through a shared MLP adapter with softplus activation. Because the adapter's input depends on the gene's DNA sequence, the token embedding changes when the sequence changes, and those changes propagate through the transformer's attention to the contextualised embeddings of other genes, allowing the model to encode sequence-driven co-regulation.
What would settle it
A direct check: take a gene with no known regulatory partners, mutate 100 bp around its TSS exactly as done for GATA4, and compare the cosine-similarity shifts of its experimentally defined targets (or of all other genes) against a random gene set; if the top-shifted genes are not the known targets and the housekeeping genes also shift, then the observed signal is not sequence-specific co-regulation. A complementary test: perturb a 100 bp region 20 kb away from the TSS that Enformer does not predict to affect expression; Bio-DTA should show no target-specific embedding shift.
Extended reading notes
Core claim
The paper's central claim is that dynamic token embeddings let a single-cell foundation model encode the impact of small DNA-sequence changes on its learned gene co-regulation networks. Concretely, Bio-DTA projects Enformer's mean-pooled 3,072-dimensional sequence embedding over 196,608 bp around each gene's TSS through a shared MLP adapter with softplus activation to form that gene's input token embedding; because the embedding depends on the DNA sequence, mutating the GATA4 TSS locally changes only GATA4's token representation, and attention propagates that change to the contextualised embeddings of other genes. In 103 fetal cardiomyocyte transcriptomes, the cosine similarity of embeddings for direct and indirect GATA4 and TBX5 targets dropped significantly after the mutation (FDR-adjusted p-values from 3.85e-09 to 6.90e-07), while housekeeping gene embeddings did not shift. For the 50 genes with the largest embedding changes, Bio-DTA achieved higher precision and recall than the in-silico-deletion Geneformer baseline and than an adapter-free BERT on direct GATA4 or TBX5 and co-bound targets, though the adapter-free BERT led on indirect TBX5 targets.
Load-bearing premise
The method assumes that averaging Enformer's 3,072-dimensional DNA embeddings over a 196,608 bp window around each gene's transcription start site preserves enough regulatory information for a single shared adapter to project faithfully into the transcriptome model's token space; if that aggregation discards the very signals the model needs, the observed embedding shifts could reflect noise or length effects rather than learned gene regulation.
Editorial extensions
If this is right
- If a single-cell foundation model can read DNA context through token adapters, then in silico mutagenesis becomes a zero-shot way to ask which genes a regulatory variant affects in a given cell type, without retraining.
- Because the adapter is shared and the method is architecture-agnostic, the same dynamic token adaptation could inject other modalities such as RNA isoforms or protein sequence into single-cell or other foundation models.
- The better recovery of direct GATA4 and TBX5 targets suggests Bio-DTA could prioritise candidate target genes for a transcription factor more accurately than deletion-based approaches, since it preserves the global input context while perturbing only the relevant gene's embedding.
- The drop in cosine similarity for targets, and its stability for housekeeping genes, offers a continuous per-gene readout of regulatory impact that could be used as a feature in variant-effect prediction.
Reading between the lines
- An implicit extension is to apply the same mutagenesis protocol to a transcription factor with no known cardiac role and to a mutated region Enformer does not predict to affect expression; if target-specific shifts persist in the second case, the signal is not sequence-specific co-regulation.
- Because Enformer is trained on the reference genome and poorly predicts personal-genome expression variation (a limitation the authors state), the zero-shot mutagenesis results may reflect chromatin-feature prediction rather than true allelic effects; fine-tuning Enformer on personal genomes, which the authors flag as future work, would settle this.
- The paper evaluates only mean-pooled single-token representations per gene; using several tokens per gene to carry the DNA sequence (also listed as future work) could preserve regulatory syntax and should be compared head-to-head on the same GATA4 benchmark.
- A sharper single-cell test of the claimed 'single-cell level' prediction would be to check whether Bio-DTA predicts heterogeneous target-gene responses across individual fetal cardiomyocytes rather than only at the population level of 103 transcriptomes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces dynamic token adaptation (DTA), an architectural mechanism that replaces fixed token embeddings in a single-cell BERT model with projections of DNA-sequence embeddings from Enformer. The resulting model, Bio-DTA, is pre-trained on a large CellXGene corpus with a masked-language-modelling objective. To evaluate it, the authors introduce random mutations in a 100 bp window around the GATA4 TSS, feed the mutated Enformer embedding into the model, and measure cosine-similarity shifts in penultimate-layer embeddings of target genes in fetal cardiomyocytes. They report significant shifts for GATA4 and TBX5 targets relative to the rest of the genome, and report precision and recall values that favour Bio-DTA over Geneformer and an adapter-free BERT on direct targets. The paper concludes that dynamic token embeddings allow sensitivity to small genetic changes and their effect on learned co-regulation networks.
Significance. If the evaluation were clean, the contribution would be useful: DTA is a general, relatively simple way to condition a single-cell foundation model on per-gene DNA sequence information, and the training scale (33 million cells, 265 datasets) is substantial. The strength of the paper is the clear architectural description and the fact that the target gene sets are independent ChIP-seq-derived labels, so the benchmark is not circular. However, the current evaluation does not yet establish the central claim: the reported metrics are embedding-similarity shifts rather than predicted expression changes, and the target-enrichment result lacks a matched negative-control perturbation. The comparison to baselines is also confounded by the different perturbation operations. The paper's potential is real, but the evidence as presented is insufficient for the strength of the claims.
major comments (4)
- [Abstract; Section 3] The abstract and Section 3 claim that the model's response to GATA4 mutagenesis is 'predicted expression changes', but the evaluation in Section 3 computes cosine similarity between contextualised gene embeddings from the penultimate layer. It never measures the model's actual expression predictions, such as token probabilities or logits over the vocabulary. This is a material overstatement: an embedding shift is not a predicted expression change, and the claim should either be reworded to describe embedding changes or supplemented with a direct readout of predicted expression.
- [Section 3, Table 1 and Figure 3] There is no negative-control perturbation. The benchmark compares GATA4 and TBX5 targets, defined by cardiac ChIP-seq and likely co-expressed with GATA4 in fetal cardiomyocytes, against 'the remainder of the genome', which is dominated by unexpressed or lowly expressed genes. Because the input to the BERT model is an expression-ordered sequence and attention can propagate any token's embedding change to co-expressed genes, the significant Wilcoxon p-values and the precision and recall enrichment could reflect a generic response to mutating an expressed gene's embedding rather than evidence of GATA4-specific regulatory knowledge. A matched control perturbation, such as mutating the TSS of an unrelated expressed transcription factor or perturbing a non-regulatory locus with matched expression, is required to attribute the result to learned co-regulation.
- [Section 3, Figure 3; Section 4] The comparison between Bio-DTA and the adapter-free BERT or Geneformer baselines is confounded by the perturbation protocol: Bio-DTA receives a small change to the GATA4 token embedding, whereas the baselines remove the GATA4 token from the input sequence entirely. This is acknowledged in Section 4, but it means the reported advantage on direct targets cannot be attributed to DTA or to the inclusion of DNA information; it may simply reflect the different magnitude and nature of the input change. A matched evaluation, such as applying the same in silico deletion to Bio-DTA or applying an embedding-level mutation to the adapter-free model, is needed before the comparison can support the stated conclusion.
- [Section 2, Figure 1] The input representation pools Enformer's 3,072-dimensional outputs over 196,608 bp by mean, and a single shared MLP adapter maps this aggregate for every gene. No ablation or analysis shows how much of the regulatory signal relevant to a 100 bp TSS mutation survives this aggregation, and no evidence rules out the possibility that the observed embedding changes are driven by sequence length or GC-content differences rather than regulatory content. At minimum, the authors should report the effect of the mutation on the aggregated Enformer embedding and include a control where the same mutation is applied to a non-regulatory sequence.
minor comments (6)
- [Section 4] The phrase 'token adaption' is a typo and should read 'token adaptation'.
- [Section 2 and throughout] Enformer is described as a 'DNA language model', but it is a convolutional sequence model; please use terminology that distinguishes it from nucleotide language models.
- [Table 1] The statistical unit is not stated; please clarify whether the Wilcoxon test compares per-cell, per-gene, or per-cell-gene cosine similarities, and report the number of genes in each target group.
- [Figure 3] The random-gene-set baseline appears to be a single size-matched sample; please report the number of random draws, seeds, and error bars, and clarify whether the top-50 selection was made per cell or after pooling cells.
- [Section 3] It is unclear whether a single mutated sequence was used or whether multiple random mutation draws were averaged; if only one was used, the result should be shown to be robust to the choice of random mutation.
- [Appendix A] The manuscript does not state whether code and model weights will be released; a reproducibility statement would be needed for a journal version.
Circularity Check
No significant circularity: independent ChIP-seq target labels benchmark an empirically probed embedding shift, with no fitted input renamed as a prediction.
full rationale
Bio-DTA's central claim is not derived from its inputs in a circular way. The target gene sets are defined by independent ChIP-seq evidence from Theodoris et al. (2023), and Bio-DTA is trained end-to-end with masked language modelling on scRNA-seq token order and fixed Enformer embeddings; the ChIP-seq labels do not appear in the loss or in the adapter construction. The in silico mutagenesis evaluation is an empirical probe: changing the GATA4 TSS sequence changes the Enformer-derived token embedding through the frozen Enformer and trained adapter, and the paper measures cosine-similarity shifts of contextualised embeddings against an unmutated reference. Enrichment of experimentally verified targets among the top-shifted genes is therefore a genuinely external benchmark rather than a quantity fitted to those labels. The paper's acknowledged limitations concern external validity, not circularity: Section 4 states that DNA language models trained on the reference genome such as Enformer 'struggle to reliably predict the direction of eQTLs and the expression variation for different individuals' (citing Huang et al. 2023 and Sasse et al. 2023), and that high-quality, well-characterized instances of perturbation with downstream co-regulation validation are rare. These are legitimate scientific caveats about generalizability and benchmark quality, not reductions of the prediction to the labels. The use of Enformer's own CAGE prediction to confirm that the mutation reduces local expression is a sanity check on the mutagenesis procedure, not a claim that Bio-DTA predicts Enformer's predictions. No fitted parameter is renamed as a prediction, and no load-bearing uniqueness theorem is imported from the authors' previous work. The possible confound of expression-level or co-expression effects in the target enrichment analysis would be a soundness or experimental-design concern, but it is not an instance of a derivation being equivalent to its inputs by construction. I therefore find no circular step and assign score 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Enformer embeddings of a 196,608 bp window around the TSS, mean-pooled over positions, contain sufficient regulatory information for each gene.
- domain assumption The BERT masked-language modeling objective on scRNA-seq gene order captures gene regulatory networks as intended.
- domain assumption ChIP-seq based target definitions from Theodoris et al. (2023) are accurate for fetal cardiomyocytes and appropriate for validating the predictions.
Cite this review
Pith. "Pith review of Multi-modal single-cell foundation models via dynamic token adaptation." pith.science (2026). https://pith.science/paper/BBL4M5KI
@misc{pith2026250413049,
author = {Pith},
title = {Pith review of: Multi-modal single-cell foundation models via dynamic token adaptation},
year = {2026},
howpublished = {\url{https://pith.science/paper/BBL4M5KI}},
note = {Machine review of arXiv:2504.13049}
}
read the original abstract
Recent advances in applying deep learning in genomics include DNA-language and single-cell foundation models. However, these models take only one data type as input. We introduce dynamic token adaptation and demonstrate how it combines these models to predict gene regulation at the single-cell level in different genetic contexts. Although the method is generalisable, we focus on an illustrative example by training an adapter from DNA-sequence embeddings to a single-cell foundation model's token embedding space. As a qualitative evaluation, we assess the impact of DNA sequence changes on the model's learned gene regulatory networks by mutating the transcriptional start site of the transcription factor GATA4 in silico, observing predicted expression changes in its target genes in fetal cardiomyocytes.
Figures
Reference graph
Works this paper leans on
-
[1]
Ledsam, Agnieszka Grabska-Barwinska, Kyle R
Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agnieszka Grabska-Barwinska, Kyle R. Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R. Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 18 0 (10): 0 1196--1203, October 2021 a . ISSN 1548-7091, 1548-7105. doi:10.1...
-
[2]
Base-resolution models of transcription-factor binding reveal soft motif syntax
Žiga Avsec, Melanie Weilert, Avanti Shrikumar, Sabrina Krueger, Amr Alexandari, Khyati Dalal, Robin Fropf, Charles McAnany, Julien Gagneur, Anshul Kundaje, and Julia Zeitlinger. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nature Genetics, 53 0 (3): 0 354--366, March 2021 b . ISSN 1061-4036, 1546-1718. doi:10.1038/s4158...
-
[3]
Unifying Vision -and- Language Tasks via Text Generation , 2021
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying Vision -and- Language Tasks via Text Generation , 2021. URL https://arxiv.org/abs/2102.02779. Version Number: 2
arXiv 2021
-
[4]
scGPT : Towards Building a Foundation Model for Single - Cell Multi -omics Using Generative AI
Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, and Bo Wang. scGPT : Towards Building a Foundation Model for Single - Cell Multi -omics Using Generative AI . preprint, Bioinformatics, May 2023. URL http://biorxiv.org/lookup/doi/10.1101/2023.04.30.538439
-
[5]
Bell, Emanuele Bezzi, Batuhan Cakir, Jim Chaffer, Signe Chambers, J
CZI Single-Cell Biology Program , Shibla Abdulla, Brian Aevermann, Pedro Assis, Seve Badajoz, Sidney M. Bell, Emanuele Bezzi, Batuhan Cakir, Jim Chaffer, Signe Chambers, J. Michael Cherry, Tiffany Chi, Jennifer Chien, Leah Dorman, Pablo Garcia-Nieto, Nayib Gloria, Mim Hastie, Daniel Hegeman, Jason Hilton, Timmy Huang, Amanda Infeld, Ana-Maria Istrate, Iva...
2023
-
[6]
BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding , May 2019
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding , May 2019. URL http://arxiv.org/abs/1810.04805. arXiv:1810.04805 [cs]
arXiv 2019
-
[7]
Shiron Drusinsky, Sean Whalen, and Katherine S. Pollard. Deep-learning prediction of gene expression from personal genomes, July 2024. URL http://biorxiv.org/lookup/doi/10.1101/2024.07.27.605449
-
[8]
Shuai, Parth Baokar, Ryan Chung, Ruchir Rastogi, Pooja Kathail, and Nilah M
Connie Huang, Richard W. Shuai, Parth Baokar, Ryan Chung, Ruchir Rastogi, Pooja Kathail, and Nilah M. Ioannidis. Personal transcriptome variation is poorly explained by current genomic deep learning models. Nature Genetics, 55 0 (12): 0 2056--2059, December 2023. ISSN 1061-4036, 1546-1718. doi:10.1038/s41588-023-01574-w. URL https://www.nature.com/article...
Show all 24 references
-
[9]
David R. Kelley. Cross-species regulatory sequence activity prediction. PLOS Computational Biology, 16 0 (7): 0 e1008050, July 2020. ISSN 1553-7358. doi:10.1371/journal.pcbi.1008050. URL https://dx.plos.org/10.1371/journal.pcbi.1008050
2020 doi
-
[10]
Kelley, Yakir A
David R. Kelley, Yakir A. Reshef, Maxwell Bileschi, David Belanger, Cory Y. McLean, and Jasper Snoek. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Research, 28 0 (5): 0 739--750, May 2018. ISSN 1088-9051, 1549-5469. do...
2018 doi
-
[11]
Knight-Schrijver, Hongorzul Davaapil, Semih Bayraktar, Alexander D
Vincent R. Knight-Schrijver, Hongorzul Davaapil, Semih Bayraktar, Alexander D. B. Ross, Kazumasa Kanemaru, James Cranley, Monika Dabrowska, Minal Patel, Krzysztof Polanski, Xiaoling He, Ludovic Vallier, Sarah Teichmann, Laure Gambardella, and Sanjay Sinha. A single-cell compar...
2022 doi
-
[12]
Dhodapkar, and David Van Dijk
Daniel Levine, Syed Asad Rizvi, Sacha Lévy, Nazreen Pallikkavaliyaveetil MohammedSheriff, Ruiming Wu, Insu Han, Zihe Zhang, Antonio Fonseca, Xingyu Chen, Sina Ghadermarzi, Amin Karbasi, Rahul M. Dhodapkar, and David Van Dijk. Cell2Sentence : Teaching Large Language Models the ...
2023 doi
-
[13]
Chuang, David Richmond, Gabriele Scalia, and Tommaso Biancalani
Sepideh Maleki, Jan-Christian Huetter, Kangway V. Chuang, David Richmond, Gabriele Scalia, and Tommaso Biancalani. Efficient Fine - Tuning of Single - Cell Foundation Models Enables Zero - Shot Molecular Perturbation Prediction , April 2025. URL http://arxiv.org/abs/2412.13478...
2025 arXiv
-
[14]
Disruption of myocardial Gata4 and Tbx5 results in defects in cardiomyocyte proliferation and atrioventricular septation
Chaitali Misra, Sheng-Wei Chang, Madhumita Basu, Nianyuan Huang, and Vidu Garg. Disruption of myocardial Gata4 and Tbx5 results in defects in cardiomyocyte proliferation and atrioventricular septation. Human Molecular Genetics, 23 0 (19): 0 5025--5035, October 2014. ISSN 0964-...
2014 doi
-
[15]
DeepSpeed : System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed : System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters . In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pp.\...
2020
-
[16]
Spiro, Shinya Tasaki, David A
Alexander Sasse, Bernard Ng, Anna E. Spiro, Shinya Tasaki, David A. Bennett, Christopher Gaiteri, Philip L. De Jager, Maria Chikina, and Sara Mostafavi. Benchmarking of deep neural networks for predicting personal gene expression from DNA sequence highlights shortcomings. Natu...
2023 doi
-
[17]
Woodruff, Stephen Young, and Kim M
Ron Schwessinger, Jacob Deasy, Rob T. Woodruff, Stephen Young, and Kim M. Branson. Single-cell gene expression prediction from DNA sequence at large contexts. preprint, Genomics, July 2023. URL http://biorxiv.org/lookup/doi/10.1101/2023.07.26.550634
2023 doi
-
[18]
Theodoris, Ling Xiao, Anant Chopra, Mark D
Christina V. Theodoris, Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R. Al Sayed, Matthew C. Hill, Helene Mantineo, Elizabeth M. Brydon, Zexian Zeng, X. Shirley Liu, and Patrick T. Ellinor. Transfer learning enables predictions in network biology. Nature, 618 0 (7965): 0 61...
2023 doi
-
[19]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...
2020 arXiv
-
[20]
scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA -seq data
Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA -seq data. Nature Machine Intelligence, 4 0 (10): 0 852--866, September 2022. IS...
2022 doi
-
[21]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[22]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[23]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[24]
Ƙ [k6Xs-g4FNʑrE 6 km k ZΏ 5G I ƬE0 .' |Hҳ B nkQrZsGșOR(Z KxEoܿs>&'FRv oUr QNk!r ,= cL c u4IwõZǡ r_Kz51 . w |9=317zILJ^ nwό1Wym aaݱK:Z C1ǩE4u tv9 ?l) `+gbDD ×»N;_NoHXZ Z Č1&
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.