Pith. sign in

REVIEW 3 major objections 5 minor 49 references

Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ctrl-DNA uses constrained reinforcement learning, with Lagrangian multipliers on off-target activities, to guide a pretrained DNA language model into generating promoters and enhancers whose predicted regulatory activity concentrates in a…

desk verdict A genuinely new constrained-RL recipe for cell-type-specific DNA design, with an evaluation that is mostly circular because the training reward and the evaluation metric are the same Enformer model. read the letter →

arxiv 2505.20578 v1 pith:UXMV6HAZ submitted 2025-05-26 cs.LG cs.AIq-bio.GN

Xingyu Chen , Shihao Ma , Runsheng Lin , Jiecong Lin , Bo Wang This is my paper
classification cs.LGcs.AIq-bio.GN
keywords regulatoryDNAdesignconstrainedreinforcementlearningcell-typespecificitycis-regulatoryelementslanguagemodelstranscriptionfactorbindingsitesLagrangianrelaxationpromoterandenhancer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ctrl-DNA sets out to solve a bottleneck in synthetic biology: writing the regulatory DNA sequences (promoters and enhancers) that switch gene expression on in one desired cell type while leaving other cell types quiet. The paper frames this as a constrained optimization problem and solves it with reinforcement learning applied to a pretrained genomic language model, replacing the usual value-function training with batch-normalized rewards and Lagrangian multipliers that penalize off-target activity. Across human enhancer and promoter datasets covering six cell lines, it reports that the method beats evolutionary, generative, and RL baselines on target-cell fitness while meeting off-target constraints at most thresholds. The payoff, if the predictions hold up in the lab, would be a general recipe for designer regulatory sequences usable in gene therapy and cell-type-targeted medicine.

What carries the argument

The carrying mechanism is constrained batch-wise relative policy optimization. For each batch of generated sequences, rewards from the target cell and each off-target cell are z-scored, and the Lagrangian multipliers $\lambda_i$ form a convex combination of these normalized advantages, so sequences that score high on-target but low off-target receive positive updates. The policy objective is a clipped importance-ratio surrogate with KL regularization toward the pretrained reference model, and each multiplier is updated by how much its cell's predicted fitness exceeds the chosen threshold $\delta_i$. A secondary reward term, the Pearson correlation between TFBS frequencies in generated and real sequences, regularizes outputs toward biologically realistic motif content, with its multiplier capped to avoid over-regularization.

What would settle it

Take the top Ctrl-DNA sequences for a target such as HepG2 and measure their actual activity by MPRA in HepG2, K562, and SK-N-SH cells; if the measured target-cell activity does not exceed the 90th percentile of real assayed enhancers while off-target activity stays at or below the constraint threshold, the claimed specificity is a reward-model artifact rather than a property of the DNA.

Watch

Extended reading notes

Core claim

The central claim is that explicit cell-type constraints can be imposed on autoregressive DNA language models during RL fine-tuning without sacrificing either target activity or sequence diversity. Formulating generation as a constrained Markov decision process, Ctrl-DNA maximizes expected fitness in the target cell under off-target fitness bounds. It estimates advantages from batch statistics rather than learned value models, and forms a Lagrangian advantage that mixes target and off-target signals through dual multipliers. On Enformer-scored human enhancer and promoter tasks, the paper reports the highest target-cell fitness among all methods while respecting off-target thresholds (for example, reward difference of 0.49 for HepG2 enhancers versus the best baseline at 0.16), and the generated sequences carry cell-type-specific transcription factor motifs such as HNF4A/HNF4G for liver and GATA1/GATA2 for erythropoietic lineages.

Load-bearing premise

Everything rests on the Enformer reward models remaining accurate for the extreme sequences the policy is pushed to produce: generated sequences score 0.77 to 0.93 predicted fitness while the 90th percentile of real assayed sequences is 0.45 to 0.59, so the optimization operates far outside the data the reward model was trained on.

Editorial extensions

If this is right

  • Users can dial a constraint threshold $\delta_i$ to trade target-cell activity against off-target suppression, giving explicit control over specificity rather than a single optimized genome.
  • Because no value networks are trained, the approach is light enough to run on one GPU in a few hours, making iterative constrained design practical for larger sequence libraries.
  • Generated sequences recover known cell-type transcription factor motifs, suggesting the optimization surface is biologically structured rather than purely numeric.
  • The method transfers across both enhancer and promoter datasets, including closely related hematopoietic cell lines where specificity is hardest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline results are measured by Enformer-predicted fitness, not by wet-lab activity, so the strongest test is whether MPRA measurements reproduce the predicted on-target/off-target gap; until then the specificity figures are in silico.
  • If reward-model extrapolation is the bottleneck, a natural extension is to interleave constrained RL with active learning that sends the most informative generated sequences to an actual MPRA round and retrains the reward model.
  • The same Lagrangian-advantage recipe could be applied to other biological design objectives with competing constraints, such as minimizing off-target CRISPR editing while maximizing on-target activity.
  • The TFBS-correlation regularizer imposes an aggregate motif-frequency match; a sharper variant would penalize the absence of specific required motifs, which could raise biological plausibility at the cost of diversity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Ctrl-DNA, a constrained reinforcement learning (RL) framework that fine-tunes an autoregressive DNA language model (HyenaDNA) to generate promoter/enhancer sequences with high predicted activity in a target cell type while keeping predicted off-target activity below user-specified thresholds. The method formulates sequence design as a constrained Markov decision process, uses batch-normalized rewards and Lagrangian multipliers to combine target and off-target advantages without training value networks, and adds a TFBS-frequency-correlation regularizer to encourage biological plausibility. The authors evaluate on human enhancer and promoter datasets using Enformer-based reward models, comparing Ctrl-DNA against evolutionary, generative, and RL baselines across six cell types and reporting target fitness, reward difference, motif correlation, and diversity.

Significance. If the results were supported by independent functional validation, Ctrl-DNA would be a practically useful contribution: it avoids value-network training, requires modest compute (about 1–2 hours on a single A100), provides code and detailed hyperparameters, and includes informative ablations (IPO, log-barrier, and λmax variants). The central limitation is that both the fitness evaluation and the motif-plausibility evaluation use the same surrogates that serve as training rewards; the reported biological claims are therefore not yet established. The methodological core—a Lagrangian, batch-wise policy-gradient approach on a pretrained DNA language model—is defensible and of interest to the sequence-design community, but the paper's current framing overstates what the evidence supports.

major comments (3)
  1. [Section 4.1 / Table 1] The fitness evaluation is circular: the Enformer reward models described in Section 4.1 are used both as the RL rewards in Section 3.2 and as the source of the reported median rewards, ΔR, and Figure 2. No independent MPRA measurements or held-out experimental assay are used to evaluate the generated sequences. This matters because the optimized sequences reach predicted fitness values of 0.77–0.93 (Appendix E, Tables 6–7), far above the 90th percentile of real assayed sequences (0.40–0.59 in Appendix B, Tables 4–5), so the policy is operating well outside the surrogate's validated range. The central claim that Ctrl-DNA 'generates high-fitness regulatory sequences' is therefore not established for actual regulatory activity. I request either an independent evaluation (for example, a held-out MPRA set or a different cell-type-specific predictor used only for evaluation) or a clear reframing of these statements as in-silico optimization results.
  2. [Section 3.3 / Section 4.1 / Table 1] The motif correlation evaluation is also circular. Section 3.3 defines RTFBS(X) = Corr(q_gen, q_real) as a training reward, where q_real is the reference TFBS frequency vector, and Section 4.1 states that the same reference motif distribution 'serve[s] as the reference motif distribution for optimization and evaluation.' The Motif Corr values in Table 1 are computed against that same q_real, so the metric is the regularized objective itself; high motif correlation is partly by construction, not independent evidence of biological plausibility. Please evaluate motif correlation against a held-out set of real sequences not used in the reward computation, or at least report the correlation gain relative to the initial untrained policy.
  3. [Section 4.2 / Appendix B] The constraint-satisfaction claim is not backed by direct violation statistics. The paper reports ΔR and the Figure 2 scatterplots but does not report, for each off-target cell type, the realized reward values Ri(X) or the fraction of generated sequences that violate Ri(X) > δi. This is important because some thresholds are close to or above the real data range: for enhancers, δ = 0.5 exceeds the 90th percentile of real activity (0.45, Appendix B Table 4), so 'satisfying the off-target constraints' may be trivially easy. Please add per-cell constraint violation rates (and, ideally, distributions of off-target rewards) for Ctrl-DNA and all baselines at each tested threshold, so the constraint enforcement claim can be assessed quantitatively.
minor comments (5)
  1. [Section 4.1] There is a typo in the evaluation-metrics paragraph: 'Additionaly' should be 'Additionally.'
  2. [Equation (5) / Algorithm 1] Equation (5) defines the Lagrangian advantage with coefficient m − Σ_i λ_i, while Algorithm 1 line 10 clips this coefficient to min(1, m − Σ_i λ_i). Please clarify which formulation is used in all experiments and how negative or greater-than-one values would be handled.
  3. [Section 3.2 / Equation (5)] The text refers to λ_i^{(j)} as a Lagrange multiplier, but the equation uses λ_i without a sample index j. Please make the notation consistent.
  4. [Section 4.1 / Section 4.3] The construction of q_real is underspecified: the reference set is defined as sequences in the top 50th percentile for the target and bottom 50th percentile for all off-target cell types, but the paper does not state how sequences meeting these criteria for multiple cell types are selected or how this set relates to the 90th-percentile re-evaluation in Section 4.3. Please specify the exact selection procedure.
  5. [Appendix C] The Bayesian optimization baseline description cites reference [40] (AdaLead) rather than the BO reference [41] (Snoek et al.); please correct the citation.

Circularity Check

2 steps flagged · score 6.0 of 10

Two evaluation claims reduce to their own optimization objectives: TFBS motif correlation is the regularizer itself, and target/off-target fitness is measured with the same Enformer reward models used for training.

  1. self definitional [Section 3.3, Algorithm 1 (lines 5–6), Section 4.1 Evaluation Metrics, Section 4.3]
    "We then quantify the similarity between generated and real sequence distributions using the Pearson correlation coefficient: RTFBS(X) = Corr(qgen, qreal) for each generated sequence X. We treat RTFBS as an additional constraint reward function... Motif Correlation is defined as the Pearson correlation between TFBS frequencies in generated sequences and real sequences... we compute motif frequency distributions, which serve as the reference motif distribution for optimization and evaluation."

    The identical correlation against the same qreal reference is both the reward the policy is trained to maximize (as an extra constraint reward, with multiplier λTFBS) and the evaluation metric used to conclude biological plausibility. High 'Motif Correlation' in Table 1 therefore measures how well the regularizer was satisfied, not an independent check of biological plausibility. The claim that Ctrl-DNA-generated sequences 'capture key cell-type-specific TFBS ... demonstrating the biological plausibility of the generated sequences' is, for this metric, true by construction: the algorithm directly optimizes the Pearson correlation of TFBS frequency vectors to the real-sequence reference distribution.

  2. fitted input called prediction [Section 4.1 (Models and Baselines; Evaluation Metrics), Eq. (1), Section 4.2]
    "max πθ EX∼πθ [R0(X)] s.t. EX∼πθ [Ri(X)] ≤ δi, ∀i ∈ {1, . . . , m}. / We adopt the Enformer architecture [38] to train cell-type-specific reward models... To assess the performance of each method, we report median rewards (Median) over the generated sequences in the final round. / Across all thresholds, Ctrl-DNA consistently achieves the highest target-cell fitness while satisfying the off-target constraints."

    The Enformer models trained on MPRA data are the reward functions in the RL objective (Eq. 1), and the same models supply the reported 'target-cell fitness' and 'off-target' numbers, since the evaluation metric is 'median rewards'. Thus the central claim that Ctrl-DNA 'generates high-fitness regulatory sequences' and 'achieves the highest target-cell fitness' is a report of how well the algorithm maximized the very surrogate oracle it was trained against, not an independent measurement of regulatory activity. Generated sequences receive predicted fitness 0.77–0.93 while the 90th percentile of assayed sequences is 0.45–0.59 (Appendix B), consistent with reward-model exploitation: the high scores are forced by optimizing the same fitted R0.

full rationale

The paper does not rely on load-bearing self-citations; the circularity is metric-level and surrogate-level. The most direct reduction is the TFBS motif correlation: Section 3.3 defines the regularizer as RTFBS(X) = Corr(qgen, qreal), and Section 4.1 defines the evaluation metric 'Motif Correlation' identically, using the same reference distribution. Consequently, the abstract's biological-plausibility evidence is partly self-referential. Second, the target and off-target fitness values are produced by the same Enformer reward models used in the RL objective (Eq. 1), and the evaluation metric is literally 'median rewards' over the final generation round. Reporting high fitness from the oracle being optimized is expected by construction and does not verify real regulatory activity; the paper even notes the optimized values far exceed the assayed 90th percentile. That said, Ctrl-DNA's algorithmic contribution—constrained Lagrangian policy gradients on a pretrained genomic LM, with comparison against many baselines under a common oracle—has independent content, and the paper honestly labels the surrogate as a reward model and discusses THP1 limitations. The partial circularity is in stretching in-silico reward optimization into 'high-fitness regulatory sequences' and 'biological plausibility' without an independent assay or a held-out predictor.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the fitted Enformer reward models (trained on MPRA data), user-chosen constraint thresholds, a hand-tuned TFBS regularizer bound, and the domain assumptions that Enformer extrapolates reliably and that TFBS correlation indicates biological plausibility. No new physical or biological entities are postulated.

free parameters (4)
  • TFBS multiplier upper bound lambda_max = 0.1
    Hand-tuned to 0.1 across all experiments; the ablation shows it affects motif correlation and occasionally target fitness (e.g., JURKAT), so central results depend on this choice.
  • Constraint thresholds delta_i = 0.3 to 0.6
    User-specified per task; results are reported across thresholds, but they are inputs, not derived, and the loose default (0.5) sits above the 90th percentile of several real sequence distributions.
  • Enformer reward model weights = trained on MPRA data
    The policy is optimized against and evaluated by these fitted models; their accuracy is the load-bearing assumption for all fitness claims.
  • KL coefficient beta and clipping epsilon = 0.2 and standard PPO default
    Standard RL hyperparameters chosen without ablation; they affect stability but are not central to the claimed contribution.
assumptions (4)
  • domain assumption Enformer reward models predict cell-type-specific CRE activity accurately for out-of-distribution sequences generated by the policy
    Invoked in Section 4.1 for reward training and Section 4.2 for evaluation; unsupported for sequences with predicted fitness far above the 90th percentile of training data.
  • domain assumption Pearson correlation between TFBS frequency vectors is a valid proxy for biological plausibility
    Section 3.3 introduces R_TFBS as regularization; Section 4.1 uses the same correlation as evaluation metric; no external validation that this correlation tracks functional activity.
  • domain assumption MPRA fitness values and the JASPAR/FIMO TFBS scanning pipeline provide reliable ground truth
    Data preprocessing follows RegLM [8]; the paper does not re-validate the MPRA measurements or motif scanning parameters.
  • standard math Policy gradient theorem and Lagrangian duality justify the primal-dual updates
    Section 3.2 builds on standard RL results; no proof of convergence for this particular batch-normalized objective is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL." pith.science (2026). https://pith.science/paper/UXMV6HAZ

@misc{pith2026250520578,
  author       = {Pith},
  title        = {Pith review of: Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXMV6HAZ}},
  note         = {Machine review of arXiv:2505.20578}
}
read the original abstract

Designing regulatory DNA sequences that achieve precise cell-type-specific gene expression is crucial for advancements in synthetic biology, gene therapy and precision medicine. Although transformer-based language models (LMs) can effectively capture patterns in regulatory DNA, their generative approaches often struggle to produce novel sequences with reliable cell-specific activity. Here, we introduce Ctrl-DNA, a novel constrained reinforcement learning (RL) framework tailored for designing regulatory DNA sequences with controllable cell-type specificity. By formulating regulatory sequence design as a biologically informed constrained optimization problem, we apply RL to autoregressive genomic LMs, enabling the models to iteratively refine sequences that maximize regulatory activity in targeted cell types while constraining off-target effects. Our evaluation on human promoters and enhancers demonstrates that Ctrl-DNA consistently outperforms existing generative and RL-based approaches, generating high-fitness regulatory sequences and achieving state-of-the-art cell-type specificity. Moreover, Ctrl-DNA-generated sequences capture key cell-type-specific transcription factor binding sites (TFBS), short DNA motifs recognized by regulatory proteins that control gene expression, demonstrating the biological plausibility of the generated sequences.

Figures

Figures reproduced from arXiv: 2505.20578 by the authors.

Figure 1
Figure 1. Overview of the Ctrl-DNA framework for controllable regulatory sequence generation. Ctrl-DNA builds on a pre-trained autoregressive DNA language model and applies constrained reinforcement learning to guide sequence generation toward high fitness in a target cell type (e.g., HepG2) while suppressing off-target fitness (e.g., K562, SK-N-SH), enabling the generation of CREs with strong cell-type specificity. To addres… view at source ↗
Figure 2
Figure 2. Pairwise fitness comparison of generated CREs highlights [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (a) Fraction of Ctrl-DNA-generated enhancers containing selected cell type-specific [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Sequence diversity scores for generated sequences on the human enhancer and promoter [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 29 canonical work pages

  1. [1]

    Machine-guided design of cell-type-targeting cis-regulatory elements

    Sager J Gosai, Rodrigo I Castro, Natalia Fuentes, John C Butts, Kousuke Mouri, Michael Alasoadura, Susan Kales, Thanh Thanh L Nguyen, Ramil R Noche, Arya S Rao, et al. Machine-guided design of cell-type-targeting cis-regulatory elements. Nature, pages 1–10, 2024

    work page 2024

  2. [2]

    A comprehensive review of retinal gene therapy

    Shannon E Boye, Sanford L Boye, Alfred S Lewin, and William W Hauswirth. A comprehensive review of retinal gene therapy. Molecular therapy, 21(3):509–519, 2013

    work page 2013

  3. [3]

    Enhanceratlas 2.0: an updated resource with enhancer annotation in 586 tissue/cell types across nine species

    Tianshun Gao and Jiang Qian. Enhanceratlas 2.0: an updated resource with enhancer annotation in 586 tissue/cell types across nine species. Nucleic acids research, 48(D1):D58–D64, 2020

    work page 2020

  4. [4]

    Designing cell-type-specific promoter sequences using conservative model-based optimization

    Aniketh Janardhan Reddy, Xinyang Geng, Michael Herschl, Sathvik Kolli, Aviral Kumar, Patrick Hsu, Sergey Levine, and Nilah Ioannidis. Designing cell-type-specific promoter sequences using conservative model-based optimization. Advances in Neural Information Processing Systems, 37:93033–93059, 2024

    work page 2024

  5. [5]

    Direct identification of hundreds of expression-modulating variants using a multiplexed reporter assay

    Ryan Tewhey, Dylan Kotliar, Daniel S Park, Brandon Liu, Sarah Winnicki, Steven K Reilly, Kristian G Andersen, Tarjei S Mikkelsen, Eric S Lander, Stephen F Schaffner, et al. Direct identification of hundreds of expression-modulating variants using a multiplexed reporter assay. Cell, 165(6):1519–1529, 2016

    work page 2016

  6. [6]

    Deciphering eukaryotic gene-regulatory logic with 100 million random promoters

    Carl G de Boer, Eeshit Dhaval Vaishnav, Ronen Sadeh, Esteban Luis Abeyta, Nir Friedman, and Aviv Regev. Deciphering eukaryotic gene-regulatory logic with 100 million random promoters. Nature biotechnology, 38(1):56–65, 2020

    work page 2020

  7. [7]

    Cell-type-directed design of synthetic enhancers

    Ibrahim I Taskiran, Katina I Spanier, Hannah Dickmänken, Niklas Kempynck, Alexandra Pan ˇcíková, Eren Can Ek¸ si, Gert Hulselmans, Joy N Ismail, Koen Theunis, Roel Vandepoel, et al. Cell-type-directed design of synthetic enhancers. Nature, 626(7997):212–220, 2024

    work page 2024

  8. [8]

    Designing realistic regulatory dna with autoregressive language models

    Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. Designing realistic regulatory dna with autoregressive language models. Genome Research, 34(9):1411–1420, 2024

    work page 2024

Show all 49 references
  1. [9]

    Regulatory dna sequence design with reinforcement learning

    Zhao Yang, Bing Su, Chuan Cao, and Ji-Rong Wen. Regulatory dna sequence design with reinforcement learning. In The Thirteenth International Conference on Learning Representations

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [11]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22–31. PMLR, 2017

    2017

  4. [12]

    Benchmarking safe exploration in deep reinforcement learning

    Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019

    1910 arXiv

  5. [13]

    The evolution, evolvability and engineering of gene regulatory dna

    Eeshit Dhaval Vaishnav, Carl G de Boer, Jennifer Molinet, Moran Yassour, Lin Fan, Xian Adiconis, Dawn A Thompson, Joshua Z Levin, Francisco A Cubillos, and Aviv Regev. The evolution, evolvability and engineering of gene regulatory dna. Nature, 603(7901):455–463, 2022

    2022

  6. [14]

    Fast activation maximization for molecular sequence design

    Johannes Linder and Georg Seelig. Fast activation maximization for molecular sequence design. BMC bioinformatics, 22:1–20, 2021

    2021

  7. [15]

    Model-based reinforcement learning for biological sequence design

    Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019

    2019

  8. [16]

    Biological sequence design with gflownets

    Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, page...

    2022

  9. [17]

    An actor-critic algorithm for constrained markov decision processes

    Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 54(3):207–213, 2005. 10

    2005

  10. [18]

    An online actor–critic algorithm with function approximation for constrained markov decision processes

    Shalabh Bhatnagar and K Lakshmanan. An online actor–critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153:688–708, 2012

    2012

  11. [19]

    Reward constrained policy optimization

    Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074, 2018

    2018 arXiv

  12. [21]

    Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints

    Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7396–7404, 2023

    2023

  13. [22]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015

    2015

  14. [23]

    Maximum a posteriori policy optimisation

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018

    2018 arXiv

  15. [24]

    Towards an understanding of default policies in multitask policy optimization

    Ted Moskovitz, Michael Arbel, Jack Parker-Holder, and Aldo Pacchiano. Towards an understanding of default policies in multitask policy optimization. In International Conference on Artificial Intelligence and Statistics, pages 10661–10686. PMLR, 2022

    2022

  16. [25]

    Confronting reward model overoptimization with constrained rlhf

    Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04373, 2023

    2023 arXiv

  17. [26]

    First order constrained optimization in policy space

    Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338–15349, 2020

    2020

  18. [27]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  19. [28]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

    1999

  20. [29]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  21. [30]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  22. [31]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

  23. [32]

    Designing dna with tunable regulatory activity using discrete diffusion

    Anirban Sarkar, Ziqi Tang, Chris Zhao, and Peter K Koo. Designing dna with tunable regulatory activity using discrete diffusion. bioRxiv, pages 2024–05, 2024

    2024

  24. [33]

    Machine-guided design of synthetic cell type-specific cis-regulatory elements

    Sager J Gosai, Rodrigo I Castro, Natalia Fuentes, John C Butts, Susan Kales, Ramil R Noche, Kousuke Mouri, Pardis C Sabeti, Steven K Reilly, and Ryan Tewhey. Machine-guided design of synthetic cell type-specific cis-regulatory elements. bioRxiv, 2023

    2023

  25. [34]

    Strategies for effectively modelling promoter-driven gene expression using transfer learning

    Aniketh Janardhan Reddy, Michael H Herschl, Xinyang Geng, Sathvik Kolli, Amy X Lu, Aviral Kumar, Patrick D Hsu, Sergey Levine, and Nilah M Ioannidis. Strategies for effectively modelling promoter-driven gene expression using transfer learning. bioRxiv, pages 2023–02, 2024

    2023

  26. [35]

    Inferring gene regulatory logic from high-throughput measurements of thousands of systematically designed promoters

    Eilon Sharon, Yael Kalma, Ayala Sharp, Tali Raveh-Sadka, Michal Levo, Danny Zeevi, Leeat Keren, Zohar Yakhini, Adina Weinberger, and Eran Segal. Inferring gene regulatory logic from high-throughput measurements of thousands of systematically designed promoters. Nature biotechn...

    2012

  27. [36]

    Jaspar 2022: the 9th release of the open-access database of transcription factor binding profiles

    Jaime A Castro-Mondragon, Rafael Riudavets-Puig, Ieva Rauluseviciute, Roza Berhanu Lemma, Laura Turchi, Romain Blanc-Mathieu, Jeremy Lucas, Paul Boddie, Aziz Khan, Nicolás Manosalva Pérez, et al. Jaspar 2022: the 9th release of the open-access database of transcription factor ...

    2022

  28. [37]

    The meme suite

    Timothy L Bailey, James Johnson, Charles E Grant, and William S Noble. The meme suite. Nucleic acids research, 43(W1):W39–W49, 2015

    2015

  29. [38]

    Effective gene expression prediction from sequence by integrating long-range interactions

    Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, ...

    2021

  30. [39]

    Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution

    Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information proc...

    2023

  31. [40]

    Adalead: A simple and robust adaptive greedy search algorithm for sequence design

    Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric D Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02141, 2020

    2010 arXiv

  32. [41]

    Practical bayesian optimization of machine learning algorithms

    Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012

    2012

  33. [42]

    The cma evolution strategy: a comparing review

    Nikolaus Hansen. The cma evolution strategy: a comparing review. Towards a new evolutionary computa- tion: Advances in the estimation of distribution algorithms, pages 75–102, 2006

    2006

  34. [43]

    Proximal exploration for model- guided protein sequence design

    Zhizhou Ren, Jiahan Li, Fan Ding, Yuan Zhou, Jianzhu Ma, and Jian Peng. Proximal exploration for model- guided protein sequence design. In International Conference on Machine Learning, pages 18520–18536. PMLR, 2022

    2022

  35. [44]

    Ipo: Interior-point policy optimization under constraints

    Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4940–4947, 2020

    2020

  36. [45]

    Reward-guided iterative refinement in diffusion models at test-time with applications to protein and dna design

    Masatoshi Uehara, Xingyu Su, Yulai Zhao, Xiner Li, Aviv Regev, Shuiwang Ji, Sergey Levine, and Tommaso Biancalani. Reward-guided iterative refinement in diffusion models at test-time with applications to protein and dna design. arXiv preprint arXiv:2502.14944, 2025

    2025 arXiv

  37. [46]

    Responsive safety in reinforcement learning by pid lagrangian methods

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133–9143. PMLR, 2020

    2020

  38. [47]

    Dirichlet diffusion score model for biological sequence generation

    Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, pages 1276–1301. PMLR, 2023

    2023

  39. [48]

    Dna-diffusion: Leveraging generative models for controlling chromatin accessibility and gene expression via synthetic regulatory elements

    Simon Senan, Aniketh Janardhan Reddy, Zach Nussbaum, Aaron Wenteler, Matei Bejan, Michael I Love, Wouter Meuleman, and Luca Pinello. Dna-diffusion: Leveraging generative models for controlling chromatin accessibility and gene expression via synthetic regulatory elements. In IC...

    2024

  40. [49]

    Latent diffusion model for dna sequence generation

    Zehui Li, Yuhao Ni, Tim August B Huygelen, Akashaditya Das, Guoxuan Xia, Guy-Bart Stan, and Yiren Zhao. Latent diffusion model for dna sequence generation. arXiv preprint arXiv:2310.06150, 2023

    2023 arXiv

  41. [50]

    Controlling gene expression with deep generative design of regulatory dna

    Jan Zrimec, Xiaozhi Fu, Azam Sheikh Muhammad, Christos Skrekas, Vykintas Jauniskis, Nora K Speicher, Christoph S Börlin, Vilhelm Verendel, Morteza Haghir Chehreghani, Devdatt Dubhashi, et al. Controlling gene expression with deep generative design of regulatory dna. Nature com...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.