{"id":"eaaebf0a-7066-4cb8-b8a7-420b783e154f","arxiv_id":"2504.13049","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors introduce dynamic token adaptation to combine DNA language models with single-cell foundation models, and show that mutating GATA4's promoter in silico shifts predicted target gene embeddings in fetal cardiomyocytes.","lead":"Dynamic token adaptation lets a single-cell model read DNA sequence embeddings for each gene instead of a fixed gene name embedding. A short in silico mutation of GATA4's DNA shifts the model's embeddings of its known target genes, suggesting the model captures regulatory links.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation lacks a negative-control perturbation; the observed target enrichment may be a generic response to mutating an expressed gene rather than evidence of learned GATA4-specific co-regulation.","rationale":"The reader's weakest assumption focuses on the input representation: mean-pooling Enformer embeddings over 196,608 bp may discard regulatory signal, and a single adapter may not generalize. This is a plausible concern, but the paper's empirical results (significant Wilcoxon tests and precision/recall above random) suggest that some signal survives the aggregation, so the mean-pooling concern is not directly fatal. The more load-bearing issue is internal validity of the evaluation: the experiment does not include a negative-control perturbation. If the observed embedding shifts are simply a generic consequence of perturbing a highly expressed gene's token in a transformer, then the claim that the model has learned GATA4-specific co-regulation networks is unsupported. The target-vs-remainder comparison is vulnerable to confounding by expression level because ChIP-seq targets in fetal cardiomyocytes are likely expressed in the same cell type, while the remainder of the genome includes many unexpressed genes. A control mutation would directly test whether the effect is specific to GATA4. This does not require changing the verdict, because the appropriate outcome is still a conditional acceptance pending additional controls, but it sharpens the condition that should be imposed.","tokens_in":7003,"tokens_out":8433,"duration_ms":84420,"concrete_test":"Repeat the Section 3 protocol mutating 100 bp around the TSS of a matched control gene (e.g., MYH6 or NKX2-5, expressed in fetal cardiomyocytes but not used to define the GATA4/TBX5 target groups), using the same 103 transcriptomes and the same top-50 selection. If this control mutation yields comparable enrichment for GATA4/TBX5 targets or for the control gene's own ChIP-seq targets, the reported specificity is a generic perturbation effect; if it yields no enrichment, the GATA4 result is specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 3 requires that the embedding shifts caused by GATA4 TSS mutagenesis are specific to GATA4's learned regulatory network. The experiment compares target genes against the remainder of the genome, but it never mutates a control gene or a control locus. Because input tokens are ordered by per-cell expression and the transformer uses attention, changing the embedding of any highly expressed, co-expressed gene can propagate to other expressed genes; the GATA4/TBX5 target set is derived from cardiac ChIP-seq and is therefore likely enriched for genes expressed in fetal cardiomyocytes. The comparison against the 'remainder of the genome' (mostly unexpressed or lowly expressed genes) is thus confounded by expression level and co-expression. The adapter-free baseline uses a different operation (token deletion) and cannot control for this. Without a matched negative-control perturbation, the significant Wilcoxon p-values and the precision/recall results could reflect generic sensitivity to perturbing an expressed token rather than learned co-regulation. This is distinct from the reader's mean-pooling concern: even if the Enformer aggregation is lossy, the evaluation still needs to rule out this confound before the embedding shifts can be attributed to regulatory knowledge. The paper's own Section 4 acknowledges the lack of direct experimental validation, but does not identify this control as necessary.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces dynamic token adaptation (DTA), an architectural mechanism that replaces fixed token embeddings in a single-cell BERT model with projections of DNA-sequence embeddings from Enformer. The resulting model, Bio-DTA, is pre-trained on a large CellXGene corpus with a masked-language-modelling objective. To evaluate it, the authors introduce random mutations in a 100 bp window around the GATA4 TSS, feed the mutated Enformer embedding into the model, and measure cosine-similarity shifts in penultimate-layer embeddings of target genes in fetal cardiomyocytes. They report significant shifts for GATA4 and TBX5 targets relative to the rest of the genome, and report precision and recall values that favour Bio-DTA over Geneformer and an adapter-free BERT on direct targets. The paper concludes that dynamic token embeddings allow sensitivity to small genetic changes and their effect on learned co-regulation networks.","tokens_in":7246,"tokens_out":5282,"duration_ms":48794,"significance":"If the evaluation were clean, the contribution would be useful: DTA is a general, relatively simple way to condition a single-cell foundation model on per-gene DNA sequence information, and the training scale (33 million cells, 265 datasets) is substantial. The strength of the paper is the clear architectural description and the fact that the target gene sets are independent ChIP-seq-derived labels, so the benchmark is not circular. However, the current evaluation does not yet establish the central claim: the reported metrics are embedding-similarity shifts rather than predicted expression changes, and the target-enrichment result lacks a matched negative-control perturbation. The comparison to baselines is also confounded by the different perturbation operations. The paper's potential is real, but the evidence as presented is insufficient for the strength of the claims.","major_comments":[{"comment":"The abstract and Section 3 claim that the model's response to GATA4 mutagenesis is 'predicted expression changes', but the evaluation in Section 3 computes cosine similarity between contextualised gene embeddings from the penultimate layer. It never measures the model's actual expression predictions, such as token probabilities or logits over the vocabulary. This is a material overstatement: an embedding shift is not a predicted expression change, and the claim should either be reworded to describe embedding changes or supplemented with a direct readout of predicted expression.","section":"Abstract; Section 3"},{"comment":"There is no negative-control perturbation. The benchmark compares GATA4 and TBX5 targets, defined by cardiac ChIP-seq and likely co-expressed with GATA4 in fetal cardiomyocytes, against 'the remainder of the genome', which is dominated by unexpressed or lowly expressed genes. Because the input to the BERT model is an expression-ordered sequence and attention can propagate any token's embedding change to co-expressed genes, the significant Wilcoxon p-values and the precision and recall enrichment could reflect a generic response to mutating an expressed gene's embedding rather than evidence of GATA4-specific regulatory knowledge. A matched control perturbation, such as mutating the TSS of an unrelated expressed transcription factor or perturbing a non-regulatory locus with matched expression, is required to attribute the result to learned co-regulation.","section":"Section 3, Table 1 and Figure 3"},{"comment":"The comparison between Bio-DTA and the adapter-free BERT or Geneformer baselines is confounded by the perturbation protocol: Bio-DTA receives a small change to the GATA4 token embedding, whereas the baselines remove the GATA4 token from the input sequence entirely. This is acknowledged in Section 4, but it means the reported advantage on direct targets cannot be attributed to DTA or to the inclusion of DNA information; it may simply reflect the different magnitude and nature of the input change. A matched evaluation, such as applying the same in silico deletion to Bio-DTA or applying an embedding-level mutation to the adapter-free model, is needed before the comparison can support the stated conclusion.","section":"Section 3, Figure 3; Section 4"},{"comment":"The input representation pools Enformer's 3,072-dimensional outputs over 196,608 bp by mean, and a single shared MLP adapter maps this aggregate for every gene. No ablation or analysis shows how much of the regulatory signal relevant to a 100 bp TSS mutation survives this aggregation, and no evidence rules out the possibility that the observed embedding changes are driven by sequence length or GC-content differences rather than regulatory content. At minimum, the authors should report the effect of the mutation on the aggregated Enformer embedding and include a control where the same mutation is applied to a non-regulatory sequence.","section":"Section 2, Figure 1"}],"minor_comments":[{"comment":"The phrase 'token adaption' is a typo and should read 'token adaptation'.","section":"Section 4"},{"comment":"Enformer is described as a 'DNA language model', but it is a convolutional sequence model; please use terminology that distinguishes it from nucleotide language models.","section":"Section 2 and throughout"},{"comment":"The statistical unit is not stated; please clarify whether the Wilcoxon test compares per-cell, per-gene, or per-cell-gene cosine similarities, and report the number of genes in each target group.","section":"Table 1"},{"comment":"The random-gene-set baseline appears to be a single size-matched sample; please report the number of random draws, seeds, and error bars, and clarify whether the top-50 selection was made per cell or after pooling cells.","section":"Figure 3"},{"comment":"It is unclear whether a single mutated sequence was used or whether multiple random mutation draws were averaged; if only one was used, the result should be shown to be robust to the choice of random mutation.","section":"Section 3"},{"comment":"The manuscript does not state whether code and model weights will be released; a reproducibility statement would be needed for a journal version.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"This is a published ICLR 2025 conference paper; the journal submission should clearly state its relationship to the conference version, and the lack of code and data release should be addressed. The scientific evaluation above stands independently of that concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: the core contribution is real. Dynamic token adaptation extends adapter-style methods from a few special tokens to every input token, and this is the first single-cell foundation model I know of that uses per-gene DNA sequence embeddings as full token embeddings. That is a new and sensible idea, and the paper is honest about its limitations.\n\nWhat it does well: the method is simple, end-to-end trainable, and large-scale (33M cells, masked LM). The GATA4 case study is a sensible proof-of-concept, and the authors correctly note they are doing zero-shot evaluation against independent ChIP-seq targets. Building an adapter-free BERT baseline with the same data and hyperparameters is a good-faith comparison. The writing is clear and the limitations paragraph in the conclusion is candid, even if it doesn't catch the biggest flaw.\n\nWhere it's soft: the evaluation has two load-bearing problems. First, the abstract says \"predicted expression changes\" but the actual metric is cosine similarity of contextualised embeddings. That is a proxy, not expression, and it should be labeled as such throughout. Second, and more important: there is no negative-control perturbation. The stress-test is on target. GATA4 is an expressed transcription factor in fetal cardiomyocytes, and its ChIP-seq targets are enriched for genes expressed in that cell type. Mutating GATA4's embedding changes a highly expressed, co-expressed token; attention and co-expression alone could propagate that shift to other expressed genes. Comparing against the \"remainder of the genome\"—mostly unexpressed genes—confounds expression level with regulatory specificity. The proper control is to mutate a matched control gene or a non-functional locus and show the target enrichment is stronger for GATA4. Without that, the Wilcoxon p-values and precision/recall could reflect generic sensitivity to perturbing an expressed token.\n\nA smaller issue: the comparison to Geneformer and the adapter-free BERT changes two things at once (mutation vs. deletion; DNA-aware vs. DNA-free input), so you can't attribute the performance difference to one factor. The mean-pooling of 196kb Enformer embeddings into a single 3,072-dim vector is also a lossy representation, though that concern is secondary to the missing control. No error bars, no released artifacts.\n\nWhom it's for: people working on multimodal single-cell models or on in-silico perturbation prediction. The idea deserves attention, but the evidence as presented supports only \"the model is sensitive to small embedding changes,\" not \"the model has learned GATA4-specific co-regulation.\"\n\nRecommendation: send it to peer review, yes. The contribution is novel enough to warrant referee time, but ask for a negative-control experiment and direct expression-level validation before publication. The path to fixing this is clear, and the paper will be stronger for it.","headline":"A genuinely new adapter idea with an evaluation that doesn't yet prove the headline claim; the missing negative-control perturbation is the main fix.","tokens_in":7765,"tokens_out":1812,"would_cite":true,"duration_ms":20846,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","92C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing DNA sequence and single-cell transcriptomes lets a model predict which genes a mutation hits.","keywords":["dynamic token adaptation","single-cell foundation model","DNA sequence embeddings","gene regulation","in silico mutagenesis","GATA4","Enformer","multi-modal learning"],"falsifier":"A direct check: take a gene with no known regulatory partners, mutate 100 bp around its TSS exactly as done for GATA4, and compare the cosine-similarity shifts of its experimentally defined targets (or of all other genes) against a random gene set; if the top-shifted genes are not the known targets and the housekeeping genes also shift, then the observed signal is not sequence-specific co-regulation. A complementary test: perturb a 100 bp region 20 kb away from the TSS that Enformer does not predict to affect expression; Bio-DTA should show no target-specific embedding shift.","tokens_in":6850,"feed_emoji":"🧬","tokens_out":6308,"duration_ms":53108,"temperature":0.7,"pith_summary":"This paper introduces dynamic token adaptation (DTA), a technique that feeds DNA-sequence information into a single-cell foundation model by replacing each gene's fixed token embedding with a projection of the DNA surrounding its transcription start site. The authors claim that this makes the model sensitive to small genetic changes: when the GATA4 promoter is mutated in silico, the model's contextualised embeddings of known GATA4 and TBX5 target genes shift significantly while housekeeping genes stay stable, and the top shifted genes recover experimentally verified targets better than Geneformer or an adapter-free BERT. If correct, this provides a zero-shot way to predict which genes a regulatory variant affects in a given cell type, moving beyond the common practice of removing the gene outright.","feed_headline":"DNA-aware single-cell model predicts mutation effects on gene networks","feed_subtitle":"Fusing DNA-sequence and single-cell transcriptome embeddings lets Bio-DTA respond to GATA4 mutations in silico.","key_machinery":"The central mechanism is dynamic token adaptation: instead of a fixed, learnable embedding per gene token (as in standard BERT or Geneformer), each gene's input embedding is computed on the fly by mean-pooling Enformer's 3,072-dimensional DNA-sequence embeddings over a 196,608 bp window around the gene's transcriptional start site and passing them through a shared MLP adapter with softplus activation. Because the adapter's input depends on the gene's DNA sequence, the token embedding changes when the sequence changes, and those changes propagate through the transformer's attention to the contextualised embeddings of other genes, allowing the model to encode sequence-driven co-regulation.","core_discovery":"The paper's central claim is that dynamic token embeddings let a single-cell foundation model encode the impact of small DNA-sequence changes on its learned gene co-regulation networks. Concretely, Bio-DTA projects Enformer's mean-pooled 3,072-dimensional sequence embedding over 196,608 bp around each gene's TSS through a shared MLP adapter with softplus activation to form that gene's input token embedding; because the embedding depends on the DNA sequence, mutating the GATA4 TSS locally changes only GATA4's token representation, and attention propagates that change to the contextualised embeddings of other genes. In 103 fetal cardiomyocyte transcriptomes, the cosine similarity of embeddings for direct and indirect GATA4 and TBX5 targets dropped significantly after the mutation (FDR-adjusted p-values from 3.85e-09 to 6.90e-07), while housekeeping gene embeddings did not shift. For the 50 genes with the largest embedding changes, Bio-DTA achieved higher precision and recall than the in-silico-deletion Geneformer baseline and than an adapter-free BERT on direct GATA4 or TBX5 and co-bound targets, though the adapter-free BERT led on indirect TBX5 targets.","pith_inferences":["An implicit extension is to apply the same mutagenesis protocol to a transcription factor with no known cardiac role and to a mutated region Enformer does not predict to affect expression; if target-specific shifts persist in the second case, the signal is not sequence-specific co-regulation.","Because Enformer is trained on the reference genome and poorly predicts personal-genome expression variation (a limitation the authors state), the zero-shot mutagenesis results may reflect chromatin-feature prediction rather than true allelic effects; fine-tuning Enformer on personal genomes, which the authors flag as future work, would settle this.","The paper evaluates only mean-pooled single-token representations per gene; using several tokens per gene to carry the DNA sequence (also listed as future work) could preserve regulatory syntax and should be compared head-to-head on the same GATA4 benchmark.","A sharper single-cell test of the claimed 'single-cell level' prediction would be to check whether Bio-DTA predicts heterogeneous target-gene responses across individual fetal cardiomyocytes rather than only at the population level of 103 transcriptomes."],"forward_implications":["If a single-cell foundation model can read DNA context through token adapters, then in silico mutagenesis becomes a zero-shot way to ask which genes a regulatory variant affects in a given cell type, without retraining.","Because the adapter is shared and the method is architecture-agnostic, the same dynamic token adaptation could inject other modalities such as RNA isoforms or protein sequence into single-cell or other foundation models.","The better recovery of direct GATA4 and TBX5 targets suggests Bio-DTA could prioritise candidate target genes for a transcription factor more accurately than deletion-based approaches, since it preserves the global input context while perturbing only the relevant gene's embedding.","The drop in cosine similarity for targets, and its stability for housekeeping genes, offers a continuous per-gene readout of regulatory impact that could be used as a feature in variant-effect prediction."],"supporting_citations":[{"why":"Supplies the Geneformer architecture, the masked-language-model training scheme, the ChIP-seq-based GATA4 and TBX5 target gene sets, and the in-silico deletion baseline.","marker":"(Theodoris et al., 2023)"},{"why":"Provides Enformer, the DNA-sequence model that produces the 3,072-dimensional embeddings used as input to the adapter.","marker":"(Avsec et al., 2021a)"},{"why":"Supplies the 103 fetal cardiomyocyte transcriptomes expressing GATA4 used in the mutagenesis evaluation.","marker":"(Knight-Schrijver et al., 2022)"},{"why":"Establishes the biological basis that GATA4 and TBX5 are co-expressed, physically interact, and share targets in cardiac morphogenesis, motivating the case study.","marker":"(Misra et al., 2014)"},{"why":"Supplies the BERT bidirectional-encoder architecture that the single-cell foundation model is based on.","marker":"(Devlin et al., 2019)"},{"why":"Provides the CELLxGENE census dataset from which the 33 million cells for pretraining were downloaded.","marker":"(CZI Single-Cell Biology Program et al., 2023)"}],"fun_headline_variants":["Dynamic token adapter links DNA to single-cell gene networks","Mutation effects predicted via DNA-aware single-cell embeddings","Token adaptation fuses DNA and single-cell data for gene networks","From DNA to expression: single-cell model with dynamic tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that averaging Enformer's 3,072-dimensional DNA embeddings over a 196,608 bp window around each gene's transcription start site preserves enough regulatory information for a single shared adapter to project faithfully into the transcriptome model's token space; if that aggregation discards the very signals the model needs, the observed embedding shifts could reflect noise or length effects rather than learned gene regulation.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic token adapter links DNA to single-cell gene networks","Mutation effects predicted via DNA-aware single-cell embeddings","Token adaptation fuses DNA and single-cell data for gene networks","From DNA to expression: single-cell model with dynamic tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000464,"raw_usage":{"total_tokens":2298,"prompt_tokens":903,"completion_tokens":1395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1330}},"tokens_in":519,"tokens_out":1395,"duration_ms":9683,"temperature":1.0,"reasoning_tokens":1330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:15:54.330383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check: take a gene with no known regulatory partners, mutate 100 bp around its TSS exactly as done for GATA4, and compare the cosine-similarity shifts of its experimentally defined targets (or of all other genes) against a random gene set; if the top-shifted genes are not the known targets and the housekeeping genes also shift, then the observed signal is not sequence-specific co-regulation. A complementary test: perturb a 100 bp region 20 kb away from the TSS that Enformer does not predict to affect expression; Bio-DTA should show no target-specific embedding shift.","supporting_citations":[{"cited_title":"Disruption of myocardial Gata4 and Tbx5 results in defects in cardiomyocyte proliferation and atrioventricular septation","cited_arxiv_id":null,"evidence_quote":"Establishes the biological basis that GATA4 and TBX5 are co-expressed, physically interact, and share targets in cardiac morphogenesis, motivating the case study."}],"review_version":1}