Pith. sign in

REVIEW 4 major objections 4 minor 62 references

Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PURE makes continuous-time RL with general function classes sample-efficient

desk verdict First CTRL sample bound with general function approximation; Eq. (5) has a fixable typo that hides the 1/√N rate, and the low-rollout guarantee leans on a constant only proven in a toy example. read the letter →

arxiv 2505.14821 v1 pith:TLHOCMWE submitted 2025-05-20 cs.LG cs.AI

Runze Zhao , Yue Yu , Adams Yiyue Zhu , Chen Yang , Dongruo Zhou This is my paper
classification cs.LGcs.AI
keywords continuous-timereinforcementlearninggeneralfunctionapproximationdistributionalEluderdimensionsamplecomplexityoptimismunderuncertaintylow-switchingRLdiffusionmodelfine-tuningneuralODEcontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Continuous-time reinforcement learning (CTRL) describes agents that choose controls while a system evolves over a continuous time horizon, but until now theoretical guarantees for CTRL with general function approximation (e.g., neural networks) were missing. This paper proposes PURE, a model-based family of algorithms that maintains optimism-based confidence sets for the unknown drift and reward, and proves the first sample-complexity bound for CTRL in this general setting: with $N$ measurements, the suboptimality gap is $\tilde{O}(\sqrt{d_{\mathcal{R}}+d_{\mathcal{F}}}\,N^{-1/2})$, where $d_{\mathcal{R}}$ and $d_{\mathcal{F}}$ are distributional Eluder dimensions. The same framework yields two practical variants: PURE$_{\text{LowSwitch}}$, which reduces policy updates from $N$ to $O(\log N\,(d_{\mathcal{F}}+d_{\mathcal{R}}))$, and PURE$_{\text{LowRollout}}$, which cuts the number of rollouts by a factor $m$ with only a mild extra term involving an independency coefficient. Experiments on diffusion-model fine-tuning and continuous control tasks show comparable reward with roughly half the training time, matching the theoretical predictions.

What carries the argument

The analysis rests on the $\ell_p$-distributional Eluder dimension of the reward and drift function classes, which measures how many distributions can disagree with a function class before evidence accumulates. Each episode builds confidence sets $\mathcal{F}_n$ and $\mathcal{R}_n$ from empirical squared-error losses, and optimism selects the policy and dynamics that maximize expected reward inside those sets. The proof chain couples Itô's lemma with Grönwall's inequality to bound how far an optimistic trajectory can deviate from the true one in mean square, then converts the confidence radii into regret sums through a chaining lemma (Theorem 5.3 of Wang et al. 2023). For the rollout-efficient variant, a new independency coefficient $C_{\mathcal{T},m}$ quantifies how close the in-episode measurement times are to i.i.d. uniform samples, and this coefficient controls the extra term in the bound.

What would settle it

Run PURE$_{\text{base}}$ on a one-dimensional Ornstein–Uhlenbeck process with known diffusion $g^*$ and a fixed measurement step $\Delta$ large enough that the Euler–Maruyama approximation $(x(t+\Delta)-x(t))/\Delta$ is visibly biased, and check whether the true drift $f^*$ stays inside every confidence set $\mathcal{F}_n$ with the claimed $1-\delta$ frequency; a violation would break the optimism argument underlying all three algorithms. A cheaper check is to measure the empirical probability over repeated runs that $f^*$ is excluded from the final confidence set as $\Delta$ varies.

Watch

Extended reading notes

Core claim

The paper establishes that continuous-time reinforcement learning with general function approximation is sample-efficient: for a stochastic differential equation model with unknown drift $f^*$ and reward $b^*$, the algorithm PURE$_{\text{base}}$, after $N$ drift and reward measurements, outputs a policy whose suboptimality gap is $O\big(T\sqrt{d_{\mathcal{R}}\beta_{\mathcal{R}}} + L T^{3/2}\sqrt{\exp(KT)}\,\sqrt{d_{\mathcal{F}}\beta_{\mathcal{F}}}\,/\sqrt{N/\log N}\big)$, with $\beta_{\mathcal{R}},\beta_{\mathcal{F}}$ logarithmic confidence radii. Treating the Eluder dimensions and horizon-dependent constants as fixed, $N = \tilde{O}(\epsilon^{-2})$ measurements suffice to reach an $\epsilon$-optimal policy. This is the first such guarantee for CTRL with general function classes, and it does not require the external calibrated epistemic-uncertainty oracle used by prior work OCoRL. The companion algorithms PURE$_{\text{LowSwitch}}$ and PURE$_{\text{LowRollout}}$ show that the sample-efficiency guarantee can be retained while cutting policy updates to $O(\log N\,(d_{\mathcal{F}}+d_{\mathcal{R}}))$ and reducing rollouts by a factor $m$, at the price of an extra $mT(d_{\mathcal{F}}+d_{\mathcal{R}})/N$ term controlled by an independency coefficient $C_{\mathcal{T},m}$ of the measurement sampler.

Load-bearing premise

The analysis assumes the agent observes unbiased Gaussian measurements of the instantaneous drift $y(t)\sim\mathcal{N}(f^*(x,u),g^{*2}/\Delta\,I)$, but in practice $y(t)$ is approximated by the Euler–Maruyama finite difference $(x(t+\Delta)-x(t))/\Delta$, and the discretization bias this introduces is not quantified; if the bias is large, $f^*$ can fall outside the confidence sets and the optimism proof collapses.

Editorial extensions

If this is right

  • If the bound holds, continuous-time model-based RL with neural-network dynamics is sample-efficient, needing only $\tilde{O}(\epsilon^{-2})$ measurements up to horizon and complexity constants.
  • PURE$_{\text{LowSwitch}}$ shows the number of costly policy-amendment steps can be reduced from $N$ to $O(\log N\,(d_{\mathcal{F}}+d_{\mathcal{R}}))$ without changing the final suboptimality gap.
  • PURE$_{\text{LowRollout}}$ shows rollouts can be batched by a factor $m$, with the only penalty an $mT(d_{\mathcal{F}}+d_{\mathcal{R}})/N$ term governed by an independency coefficient that is small when measurement times are near-uniform.
  • The theoretical predictions match the empirical gains: roughly half the training time on diffusion-model fine-tuning and continuous control while keeping final rewards.
  • The framework extends the distributional-Eluder-based analysis, previously applied to discrete-time RL, to SDE-driven environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct corollary the authors leave implicit: for any sampler with $C_{\mathcal{T},m}=O(1)$, the rollout-efficient variant has an asymptotically identical $\epsilon^{-2}$ sample complexity, so the real cost of batching measurements appears only in the finite-horizon constants.
  • If the unquantified Euler–Maruyama bias were bounded (for example by a Lipschitz-plus-second-order condition on $f^*$), the same confidence-set machinery would go through with an additive bias term, making the algorithm more robust in practice.
  • The independency coefficient is a new object that could be computed or bounded for other measurement strategies, such as adaptive or event-triggered sampling, connecting to the active-observing line of work in CTRL.
  • Because the horizon dependence is $\exp(T)$, the result is meaningful mainly for short-horizon tasks; closing this gap would require a different argument than the Grönwall-based trajectory deviation bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PURE, a family of model-based continuous-time reinforcement learning (CTRL) algorithms for general function approximation. PUREbase maintains optimism-based confidence sets for the unknown drift f* and reward b*, collects N measurements, and outputs a uniformly random policy from the N episodes. The authors claim a suboptimality gap of Õ(√(d_R+d_F) N^{-1/2}) up to horizon-dependent factors, where d_R and d_F are distributional Eluder dimensions of the squared-error reward and drift classes. PURELowSwitch reduces the number of policy updates to O(log N (d_R+d_F)), and PURELowRollout reduces the number of rollouts by a factor m at the cost of an added 'independence coefficient' C_{T,m}. The paper includes a full proof appendix and experiments on diffusion-model fine-tuning and continuous control tasks.

Significance. If the stated results hold, this is a substantial contribution: it provides what appears to be the first finite-sample guarantee for stochastic CTRL with general function approximation, using a relatively clean optimism-based analysis with distributional Eluder dimension. The appendix contains detailed proofs, including high-probability confidence-set lemmas, a trajectory-deviation bound via Itô's lemma and Grönwall's inequality, and an Eluder chaining argument. The paper also ships reproducible code and experiments on two application domains, which strengthens its practical relevance. The main reservations are the inconsistent display of Theorem 4.5, the unquantified Euler–Maruyama bias, and the conditional nature of the C_{T,m} bounds; these need to be addressed before the headline claims are fully established.

major comments (4)
  1. [Theorem 4.5, Eq. (5), and Appendix B] The displayed suboptimality gap in Eq. (5) of Theorem 4.5 does not match the proof. Theorem B.5 proves a cumulative regret bound of O(T√(N d_R β_R log N) + L T^{3/2}√(N e^{KT} d_F β_F log N)), and since Algorithm 1 outputs (π_n,q_n) with n uniform on [N], the suboptimality gap must be the cumulative regret divided by N, i.e., O(T√(d_R β_R log N / N) + L T^{3/2}√(e^{KT} d_F β_F log N / N)). In the printed Eq. (5), the factor √(N/log N) appears only in the denominator of the second term; the reward term T√(d_R β_R) is independent of N, so the printed bound does not vanish as N→∞ and cannot imply the stated sample complexity N = Õ(ε^{-2}) immediately after the theorem. Because that sample-complexity formula and the abstract's N^{-1/2} rate both use the correct scaling, this appears to be a typo, but Theorem 4.5, Eq. (5) must be corrected before the headline claim is formally established.
  2. [Section 3, Remark 3.3, and Section 7] The theoretical guarantees are proven for the unbiased instantaneous-drift oracle y(t) ~ N(f*(x,u), g*^2/Δ I) defined in Section 3, while Remark 3.3 and the experiments replace it with the Euler–Maruyama difference (x(t+Δ)-x(t))/Δ. As the authors note in Section 7, the discretization bias is unquantified. If the drift observations are biased, f* may not belong to the confidence sets F_n of Eq. (3), and the optimism argument in the proof of Theorem B.5 breaks. Section 7 should either provide a bound on the bias in terms of Δ and L_f, or explicitly restrict the statements of Theorems 4.5, 5.1, and 5.6 to the oracle measurement model.
  3. [Definition 5.4, Proposition 5.5, and Theorem 5.6] Definition 5.4 introduces the independence coefficient C_{T,m}, and Theorem 5.6 states the PURELowRollout suboptimality gap in terms of it. Proposition 5.5 bounds C_{T,m} only for a specific one-dimensional Ornstein–Uhlenbeck system with a singleton drift class and a one-parameter reward class. No general condition is given under which C_{T,m}=O(1) for the general function classes featured in Theorem 4.5, so the claim in Section 1 and Remark 5.3 that a sampler can be chosen to achieve C_{T,m}=O(1) is not supported in the general setting. The theorem is valid as a conditional statement, but the paper should state that the computational-efficiency gain of PURELowRollout is established only under an unverified condition on the sampling strategy.
  4. [Algorithm 1, line 3 and Algorithm 3, line 3] Algorithm 1 (line 3) and Algorithm 3 (line 3) require solving the optimistic planning problem arg max_{π∈Π,q∈Q,f∈F_n,b∈R_n} R(π,q,f,b). For general function classes this is a nonconvex optimization problem, and the paper neither assumes an oracle nor discusses its computational cost. Since the title and contributions emphasize computationally efficient CTRL, the statement should either explicitly assume access to an exact planning oracle or analyze an approximate solver; without this, the 'computationally efficient' claim covers only the number of policy updates and rollouts, not the total computation.
minor comments (4)
  1. [Abstract] The abstract's stated rate Õ(√(d_R+d_F) N^{-1/2}) omits the horizon factor L T^{3/2}√(exp(KT)) present in Theorem 4.5; please state the full dependence or explicitly say T is fixed.
  2. [Theorem 4.5 and Theorem 5.6 displays] The typesetting of Eq. (5) and the display in Theorem 5.6 is garbled in the preprint; please ensure the formulas render correctly in the final version, as the current ambiguity is the likely source of the inconsistency in the reward term.
  3. [Remark 3.1] Remark 3.1 refers to equation '(5.4)' when comparing the diffusion-model backward process (2) to the general SDE (1); this appears to be a leftover from an earlier draft and should be corrected.
  4. [Proof of Proposition 5.5 and Lemma B.2] The proof of Proposition 5.5 contains a typo in the final sentence ('C T,mR' should be C_{T,m,R}), and Lemma B.2 cites 'Lemma 1,5, Russo and Van Roy 2013' where it should cite Lemmas 1 and 5 of Russo and Van Roy (2013); please fix these in the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all confidence radii are set analytically, the complexity measures are problem-dependent dimensions of fixed function classes, and the load-bearing chaining lemmas are cited from external work.

full rationale

The paper's derivation chain is not circular. The quantities that appear in the main regret bound are the distributional Eluder dimensions d_R and d_F of the squared-error classes { (b-b*)^2 : b in R } and { ||f-f*||_2^2 : f in F }, which are problem-dependent properties of the fixed function classes and the unknown ground truth, not parameters fitted to the algorithm's data. The confidence radii beta_R and beta_F are set analytically from covering numbers and the failure probability (Theorem 4.5, Theorem B.4), not from the collected measurements. The key chaining step that converts confidence radii into sums of squared errors is quoted from Theorem 5.3 of Wang et al. 2023, and Proposition 4.4 relies on Lemma 5.4 of Wang et al. 2023 and Proposition 29 of Jin et al. 2021, all external to the present authors. The low-switching and low-rollout analyses explicitly borrow their main idea from Xiong et al. 2023, also external. The only references involving a current coauthor (Wang et al. 2021 and He et al. 2023) appear in the related-work discussion of low-switching linear RL and are not load-bearing for the theorems. The independence coefficient C_{T,m} is defined as a supremum over policies and initial distributions of a ratio of squared-error expectations; it is a problem-dependent constant, not fitted, and its bound in Proposition 5.5 is proved directly for an OU example. There is a displayed-equation inconsistency in Theorem 4.5 (Eq. 5): the reward term lacks a 1/sqrt(N) factor that the proof's cumulative-regret bound and the subsequent N = O-tilde(epsilon^{-2}) corollary require. That is a correctness or typographical concern, not a circularity, because the proof and corollary are not derived by assuming the printed bound. No circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central theoretical claims are built on standard RL assumptions (bounded, Lipschitz functions, known diffusion, exact planning oracle) plus an unverified independence coefficient for the rollout-reduction variant. The experimental free parameters (λ, η_base, m) are hand-chosen and do not affect the proofs.

free parameters (3)
  • λ (sampler temperature) = 6
    Chosen by hand for PURESEIKO's geometric time sampler; not derived from theory and not used in the theoretical guarantees.
  • η_base (batch growth factor) = 2
    Set for both PURESEIKO and PUREENODE experiments; empirical choice.
  • m (measurement frequency) = 4 (diffusion), 250 (control)
    Set per environment; the theory allows any m but the bound degrades with the m/N term.
assumptions (5)
  • domain assumption Assumption 4.2: boundedness and Lipschitz continuity of f, b, g, π
    Used throughout the trajectory deviation and regret analysis (Lemma B.1 and Theorem B.5).
  • domain assumption Known diffusion coefficient g* (Remark 3.2)
    The algorithm needs g* to construct drift measurements and confidence sets.
  • domain assumption Measurement model: y ~ N(f*, g*^2/Δ) and r ~ N(b*, 1) (Section 3)
    Core to the confidence-set construction; if the drift observations are biased, optimism fails.
  • ad hoc to paper Optimistic planning oracle: arg max over π,q,f∈F_n,b∈R_n R(π,q,f,b) is solved exactly (Algorithm 1, line 3)
    Computational cost of this step is not analyzed; assumed feasible for the theory.
  • ad hoc to paper C_{T,m} is bounded (Theorem 5.6)
    The rollout-reduction guarantee depends on the independence coefficient being small; only proved for a 1D OU process (Proposition 5.5).
invented entities (1)
  • Independence coefficient C_{T,m}
    purpose: Quantifies how much in-episode measurements deviate from i.i.d. uniform samples; controls the PURELowRollout regret bound.
    Only bounded for a toy one-dimensional OU process; no general bound or empirical estimator is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation." pith.science (2026). https://pith.science/paper/TLHOCMWE

@misc{pith2026250514821,
  author       = {Pith},
  title        = {Pith review of: Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TLHOCMWE}},
  note         = {Machine review of arXiv:2505.14821}
}
abstract

Continuous-time reinforcement learning (CTRL) provides a principled framework for sequential decision-making in environments where interactions evolve continuously over time. Despite its empirical success, the theoretical understanding of CTRL remains limited, especially in settings with general function approximation. In this work, we propose a model-based CTRL algorithm that achieves both sample and computational efficiency. Our approach leverages optimism-based confidence sets to establish the first sample complexity guarantee for CTRL with general function approximation, showing that a near-optimal policy can be learned with a suboptimality gap of $\tilde{O}(\sqrt{d_{\mathcal{R}} + d_{\mathcal{F}}}N^{-1/2})$ using $N$ measurements, where $d_{\mathcal{R}}$ and $d_{\mathcal{F}}$ denote the distributional Eluder dimensions of the reward and dynamic functions, respectively, capturing the complexity of general function approximation in reinforcement learning. Moreover, we introduce structured policy updates and an alternative measurement strategy that significantly reduce the number of policy updates and rollouts while maintaining competitive sample efficiency. We implemented experiments to backup our proposed algorithms on continuous control tasks and diffusion model fine-tuning, demonstrating comparable performance with significantly fewer policy updates and rollouts.

Figures

Figures reproduced from arXiv: 2505.14821 by the authors.

Figure 1
Figure 1. Summary of the experiment for fine-tuning Diffusion Models. 1a presents a comparison of aesthetic scores for [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Summary of continuous-time control experiments, PURE [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison between SEIKO and our PURESEIKO approach, with aesthetic scores listed below each image [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Summary of the ablation studies for continuous-time control in the Acrobot environment. Figures 4a and 4b [PITH_FULL_IMAGE:figures/full_fig_p028_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 34 canonical work pages

  1. [1]

    Improved algorithms for linear stochastic bandits

    Yasin Abbasi-Yadkori, D \'a vid P \'a l, and Csaba Szepesv \'a ri. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011

    2011

  2. [2]

    Efficient optimistic exploration in linear-quadratic regulators via lagrangian relaxation

    Marc Abeille and Alessandro Lazaric. Efficient optimistic exploration in linear-quadratic regulators via lagrangian relaxation. In International Conference on Machine Learning, pages 23--31. PMLR, 2020

    work page 2020

  3. [3]

    Stochastic interpolants: A unifying framework for flows and diffusions

    Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023

    arXiv 2023

  4. [4]

    Finite-time analysis of the multiarmed bandit problem, 2002

    P Auer. Finite-time analysis of the multiarmed bandit problem, 2002

    2002

  5. [5]

    Provably efficient q-learning with low switching cost

    Yu Bai, Tengyang Xie, Nan Jiang, and Yu-Xiang Wang. Provably efficient q-learning with low switching cost. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  6. [6]

    Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon

    Matteo Basei, Xin Guo, Anran Hu, and Yufei Zhang. Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research, 23 0 (178): 0 1--34, 2022

    work page 2022

  7. [7]

    The stability of solutions of linear differential equations

    Richard Bellman. The stability of solutions of linear differential equations . Duke Mathematical Journal, 10 0 (4): 0 643 -- 647, 1943

    work page 1943

  8. [8]

    Training diffusion models with reinforcement learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023

    arXiv 2023

Show all 62 references
  1. [9]

    Variational inference: A review for statisticians

    David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112 0 (518): 0 859--877, 2017

    2017

  2. [10]

    Openai gym

    G Brockman. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  3. [11]

    Caines and David Levanony

    Peter E. Caines and David Levanony. Stochastic -optimal linear quadratic adaptation: An alternating controls policy. SIAM Journal on Control and Optimization, 57 0 (2): 0 1094--1126, 2019

    2019

  4. [12]

    Online learning with switching costs and other adaptive adversaries

    Nicolo Cesa-Bianchi, Ofer Dekel, and Ohad Shamir. Online learning with switching costs and other adaptive adversaries. Advances in Neural Information Processing Systems, 26, 2013

    2013

  5. [13]

    Neural ordinary differential equations

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  6. [14]

    Online linear quadratic control

    Alon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar. Online linear quadratic control. In International Conference on Machine Learning, pages 1029--1038. PMLR, 2018

    2018

  7. [15]

    Reinforcement learning in continuous time and space

    Kenji Doya. Reinforcement learning in continuous time and space. Neural Computation, 12 0 (1): 0 219--245, 2000

    2000

  8. [16]

    A provably efficient algorithm for linear markov decision process with low switching cost

    Minbo Gao, Tianle Xie, Simon S Du, and Lin F Yang. A provably efficient algorithm for linear markov decision process with low switching cost. arXiv preprint arXiv:2101.00494, 2021

    2021 arXiv

  9. [17]

    Hamiltonian neural networks

    Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    Nearly minimax optimal reinforcement learning for linear markov decision processes

    Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear markov decision processes. In International Conference on Machine Learning, pages 12790--12822. PMLR, 2023

    2023

  11. [19]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

    2020

  12. [20]

    Active observing in continuous-time control

    Samuel Holt, Alihan H \"u y \"u k, and Mihaela van der Schaar. Active observing in continuous-time control. Advances in Neural Information Processing Systems, 36, 2024

    2024

  13. [21]

    Towards deployment-efficient reinforcement learning: Lower bound and optimality

    Jiawei Huang, Jinglin Chen, Li Zhao, Tao Qin, Nan Jiang, and Tie-Yan Liu. Towards deployment-efficient reinforcement learning: Lower bound and optimality. arXiv preprint arXiv:2202.06450, 2022

    2022 arXiv

  14. [22]

    Sublinear regret for a class of continuous-time linear--quadratic reinforcement learning problems

    Yilie Huang, Yanwei Jia, and Xun Yu Zhou. Sublinear regret for a class of continuous-time linear--quadratic reinforcement learning problems. arXiv preprint arXiv:2407.17226, 2024

    2024 arXiv

  15. [23]

    Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms

    Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34: 0 13406--13418, 2021

    2021

  16. [24]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  17. [25]

    Online sub-sampling for reinforcement learning with general function approximation

    Dingwen Kong, Ruslan Salakhutdinov, Ruosong Wang, and Lin F Yang. Online sub-sampling for reinforcement learning with general function approximation. arXiv preprint arXiv:2106.07203, 2021

    2021 arXiv

  18. [26]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  19. [27]

    I ^2 sb: Image-to-image schr \"o dinger bridge

    Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. I ^2 sb: Image-to-image schr \"o dinger bridge. In International Conference on Machine Learning, pages 22042--22062. PMLR, 2023

    2023

  20. [28]

    Let us build bridges: Understanding and extending diffusion generative models

    Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208.14699, 2022 a

    2022 arXiv

  21. [29]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022 b

    2022

  22. [30]

    Value iteration in continuous actions, states and time

    Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, and Animesh Garg. Value iteration in continuous actions, states and time. In International Conference on Machine Learning, pages 7224--7234. PMLR, 2021

    2021

  23. [31]

    Numerical solution of stochastic differential equations with jumps in finance, volume 64

    Eckhard Platen and Nicola Bruti-Liberati. Numerical solution of stochastic differential equations with jumps in finance, volume 64. Springer Science & Business Media, 2010

    2010

  24. [32]

    Aligning text-to-image diffusion models with reward backpropagation

    Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023

    2023 arXiv

  25. [33]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  26. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684--10695, 2022

    2022

  27. [35]

    Linear bandits with limited adaptivity and learning distributional optimal design

    Yufei Ruan, Jiaqi Yang, and Yuan Zhou. Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 74--87, 2021

    2021

  28. [36]

    Eluder dimension and the sample complexity of optimistic exploration

    Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013

    2013

  29. [37]

    Schuhmann

    C. Schuhmann. Laion aesthetic predictor. https://laion.ai/blog/laion-aesthetics/, 2022. Accessed: 2024-09-29

    2022

  30. [38]

    LAION -5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. L...

    2022

  31. [39]

    Diffusion schr \"o dinger bridge matching

    Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schr \"o dinger bridge matching. Advances in Neural Information Processing Systems, 36, 2024

    2024

  32. [40]

    Online reinforcement learning in stochastic continuous-time systems

    Mohamad Kazem Shirani Faradonbeh and Mohamad Sadegh Shirani Faradonbeh. Online reinforcement learning in stochastic continuous-time systems. In Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 612--656. PM...

    2023

  33. [41]

    Naive exploration is optimal for online lqr

    Max Simchowitz and Dylan Foster. Naive exploration is optimal for online lqr. In International Conference on Machine Learning, pages 8937--8948. PMLR, 2020

    2020

  34. [42]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256--2265. PMLR, 2015

    2015

  35. [43]

    Aligned diffusion schr \"o dinger bridges

    Vignesh Ram Somnath, Matteo Pariset, Ya-Ping Hsieh, Maria Rodriguez Martinez, Andreas Krause, and Charlotte Bunne. Aligned diffusion schr \"o dinger bridges. In Uncertainty in Artificial Intelligence, pages 1985--1995. PMLR, 2023

    1985

  36. [44]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020 a

    2010 arXiv

  37. [45]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020 b

    2011 arXiv

  38. [46]

    Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning

    Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning. SIAM Journal on Control and Optimization, 62 0 (1): 0 135--166, 2024

    2024

  39. [47]

    Improving and generalizing flow-based generative models with minibatch optimal transport

    Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2302.00482, 2023

    2023 arXiv

  40. [48]

    Efficient exploration in continuous-time model-based reinforcement learning

    Lenart Treven, Jonas H \"u botter, Florian Dorfler, and Andreas Krause. Efficient exploration in continuous-time model-based reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  41. [49]

    When to sense and control? a time-adaptive approach for continuous-time rl

    Lenart Treven, Bhavya Sukhija, Yarden As, Florian D \"o rfler, and Andreas Krause. When to sense and control? a time-adaptive approach for continuous-time rl. arXiv preprint arXiv:2406.01163, 2024 b

    2024 arXiv

  42. [50]

    Feedback efficient online fine-tuning of diffusion models

    Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Sergey Levine, and Tommaso Biancalani. Feedback efficient online fine-tuning of diffusion models. arXiv preprint arXiv:2402.16359, 2024

    2024 arXiv

  43. [51]

    Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems

    Draguna Vrabie and Frank Lewis. Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks, 22 0 (3): 0 237--246, 2009

    2009

  44. [52]

    The benefits of being distributional: Small-loss bounds for reinforcement learning

    Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, and Wen Sun. The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023

    2023

  45. [53]

    Provably efficient reinforcement learning with linear function approximation under adaptivity constraints

    Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems, 34: 0 13524--13536, 2021

    2021

  46. [54]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019

    2019

  47. [55]

    Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning

    Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ...

    2023

  48. [56]

    A general framework for sequential decision-making under adaptivity constraints

    Nuoya Xiong, Zhaoran Wang, and Zhuoran Yang. A general framework for sequential decision-making under adaptivity constraints. In Forty-first International Conference on Machine Learning, 2023

    2023

  49. [57]

    a hdesm \

    Cagatay Yildiz, Markus Heinonen, and Harri L \"a hdesm \"a ki. Continuous-time model-based reinforcement learning. In International Conference on Machine Learning, pages 12009--12018. PMLR, 2021

    2021

  50. [58]

    Censored sampling of diffusion models using 3 minutes of human feedback

    TaeHo Yoon, Kibeom Myoung, Keon Lee, Jaewoong Cho, Albert No, and Ernest Ryu. Censored sampling of diffusion models using 3 minutes of human feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  51. [59]

    Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon

    Zihan Zhang, Xiangyang Ji, and Simon Du. Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon. In Conference on Learning Theory, pages 4528--4531. PMLR, 2021 a

    2021

  52. [60]

    Improved variance-aware confidence sets for linear bandits and linear mixture mdp

    Zihan Zhang, Jiaqi Yang, Xiangyang Ji, and Simon S Du. Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems, 34: 0 4342--4355, 2021 b

    2021

  53. [61]

    A nearly optimal and low-switching algorithm for reinforcement learning with general function approximation

    Heyang Zhao, Jiafan He, and Quanquan Gu. A nearly optimal and low-switching algorithm for reinforcement learning with general function approximation. arXiv preprint arXiv:2311.15238, 2023

    2023

  54. [62]

    Unsupervised learning of lagrangian dynamics from images for prediction and control

    Yaofeng Desmond Zhong and Naomi Leonard. Unsupervised learning of lagrangian dynamics from images for prediction and control. Advances in Neural Information Processing Systems, 33: 0 10741--10752, 2020

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.