Pith. sign in

REVIEW 5 major objections 6 minor 58 references

Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Within a dataset's convex hull and its small neighborhood, smoothing OOD action estimates toward in-sample neighbors provably shrinks error to the true Q-value; the resulting algorithm tops D4RL benchmarks.

desk verdict SQOG is a fast, empirically strong offline RL method whose theoretical guarantees rest on an unproven regularity assumption about the true Q-function and whose practical algorithm deviates from the proposed operator in a way the paper acknowledges but does not resolve. read the letter →

arxiv 2506.08417 v1 pith:R26Q7EGU submitted 2025-06-10 cs.LG cs.AI

Qingmao Yao , Zhichao Lei , Tianyuan Chen , Ziyue Yuan , Xuefan Chen , Jianxiang Liu , Faguo Wu , Xiao Zhang This is my paper
classification cs.LGcs.AI
keywords offlinereinforcementlearningout-of-distributionactionsQ-valueoverestimationconvexhullsmoothBellmanoperatorQ-functiongeneralizationover-constraintD4RLbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning fails when the learned policy proposes actions the dataset never contains: their Q-values get overestimated, and the usual cures — penalizing or forbidding out-of-distribution (OOD) actions — end up under-estimating nearby actions too, a problem the paper calls over-constraint. This paper tries to show that a middle ground exists: OOD actions inside the convex hull of the dataset's state-action pairs, plus a thin neighborhood around that hull (together, the CHN), can be evaluated safely by smoothing their Q-values toward those of nearby in-sample actions. The paper defines the Smooth Bellman Operator (SBO) that performs this smoothing and proves that each application moves OOD estimates strictly closer to the true Q-values while disturbing in-sample values only negligibly. The resulting algorithm, SQOG, adds a noise-perturbed generalization loss onto a standard actor-critic and reports the best average scores among compared methods on D4RL's Mujoco, Maze2d, and Adroit suites at about the computational cost of TD3+BC and far below conservative rivals. A sympathetic reader would care because the paper challenges the default assumption that every OOD region is dangerous: it claims one precisely delimited slice of OOD space is actually exploitable.

What carries the argument

The CHN (Convex Hull and its Neighborhood), the union of the convex hull of the dataset's state-action points with the layer of points outside it within distance $r \le B$ where $B$ is the convex hull's diameter, is the region where the paper claims Q-generalization is safe; its compactness yields uniform continuity (Proposition 2) and its bounded internal distances keep the NTK-based bounds finite (Proposition 1). The Smooth Bellman Operator (SBO), $\tilde{\mathcal{B}}^\pi = G_1 \hat{\mathcal{B}}_2^\pi$, keeps the empirical Bellman backup for in-sample actions and, for an OOD action inside the CHN, replaces the target with the Q-value of a neighboring in-sample action; it is the mechanism that carries the argument, with Theorem 3 showing each step contracts OOD estimates toward the true value and Proposition 4 showing the operator is a $\gamma$-contraction in $L^\infty$. The whole edifice rests on Lemma 2's NTK kernel-map smoothness bound $\|Q(x) - Q(x')\| \le C(\sqrt{\min(\|x\|,\|x'\|)}\sqrt{d} + 2d)$, which bounds Q-value differences by input distance, and on the practical trick of generating OOD actions as $a_{\mathrm{ood}} = a_{\mathrm{in}} + \eta$ so the generalization loss costs almost nothing.

What would settle it

Take a small environment with exactly computable returns (the Inverted Double Pendulum sanity check already does this), compute $Q^\pi$ by Monte Carlo on a dense grid of actions inside the CHN, and run SQOG with its real finite-width critic: if for any OOD action $\|Q^\pi(s,a) - Q_{k+1}(s,a)\|$ is not strictly smaller than $\|Q^\pi(s,a) - Q_k(s,a)\|$ at some training step, Theorem 3's contraction fails as stated. A second, independent check targets Proposition 1: measure $\max_{x \in \mathrm{CHN}} \|Q_\theta(x) - Q_\theta(\mathrm{Proj}_D(x))\|$ on a finite-width ReLU critic and compare it to the bound $C(\sqrt{\min(\|x\|, \|\mathrm{Proj}_D(x)\|)}\sqrt{d} + 2d)$; if the observed gap exceeds the bound substantially at the neighborhood radius $r$ the algorithm actually uses, the safety guarantee that justifies smoothing does not hold for the deployed model.

Watch

Extended reading notes

Core claim

Within the CHN — the convex hull of the dataset's $(s,a)$ pairs together with its external neighborhood of radius $r$ — the paper claims two safety properties: under the NTK regime the Q-value at any point in the CHN differs from the Q-value of its nearest dataset point by a bound controlled by the point-to-dataset distance (Proposition 1), and the Q-function is uniformly continuous on the CHN (Proposition 2). The Smooth Bellman Operator $\tilde{\mathcal{B}}^\pi = G_1 \hat{\mathcal{B}}_2^\pi$ then replaces the backup for an OOD action inside the CHN with the current value of a neighboring in-sample action, leaving in-sample backups untouched (Definition 2). Theorems 1–3 argue that the empirical Bellman operator is close to the true one for in-sample data, that $G_1$ changes in-sample evaluation negligibly, and that for an OOD action in the CHN whose estimate deviates from its neighbor's by more than $\varepsilon$, one gradient step through the SBO strictly reduces the distance to the true Q-value, $\|Q^\pi(s,a) - Q_{k+1}(s,a)\| < \|Q^\pi(s,a) - Q_k(s,a)\|$, while already-small deviations stay within $2\varepsilon$ afterwards. SQOG instantiates this as the extra MSE loss $(Q_\theta(s, a+\eta) - \bar{Q}_\theta(s,a))^2$ on noise-perturbed in-sample actions with the neighbor value detached, and the paper reports that this both repairs Q-estimates in its sanity check and raises average scores on D4RL's Mujoco, Maze2d, and Adroit suites. The paper's position is that generalizing the Q-function inside the CHN is safe and beneficial, while generalizing beyond it remains an open challenge.

Load-bearing premise

Every theoretical guarantee — the CHN safety bounds (Proposition 1), the closeness of the empirical Bellman operator (Theorem 1), and the OOD contraction (Theorem 3) — assumes the Q-function is an infinitely wide two-layer ReLU network trained with an infinitesimally small learning rate (the NTK regime, Assumption 1 in Appendix A), whereas the algorithm that is actually evaluated and released trains finite-width networks with Adam; if finite-width networks violate the smoothness bound that this assumption yields, the paper's guarantees are not established for the deployed setting.

Editorial extensions

If this is right

  • If Theorem 3's contraction holds, the critic's OOD Q-values inside the CHN move toward the true values during training, so the actor is no longer steered by systematically wrong estimates when it considers slightly-out-of-distribution actions with genuinely higher returns.
  • Because Theorem 2 bounds the effect on in-sample evaluation, the smoothing term can be appended to any policy-constraint offline RL method without undoing the in-sample accuracy the constraint protects; the paper demonstrates this by adding SBO to BRAC and reporting large improvements.
  • The $\gamma$-contraction property (Proposition 4) implies the SBO has a unique fixed point, so the OOD smoothing introduces no new source of divergence in the idealized setting.
  • If the CHN safety guarantees hold, then methods that penalize all OOD actions indiscriminately are needlessly conservative; the correct boundary is the CHN itself, and only regions beyond it should be constrained.
  • SQOG's generalization loss reuses in-sample actions plus noise, so the accuracy gain on D4RL comes with almost no added computation, avoiding the generative-model overhead that makes the strongest comparable method (MCQ) about 20 times slower.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the CHN argument suggests a general design principle — constrain by geometry, not by 'never-taken' status: the risky set is what lies more than a dataset-diameter away from the behavior policy's data, and other mechanisms (pessimistic bonuses, ensembles, implicit regularization) could be re-aimed at the complement of the CHN and lose less performance.
  • My inference: Theorem 3's contraction is proven for gradient descent with an infinitesimal learning rate, where $Q_{k+1} = Q_k + 2\alpha[Q_{\text{neighbor}} - Q_k]$ cannot overshoot; with the finite learning rate and Adam actually used, large gaps can overshoot, so the practical guarantee is about direction of movement rather than monotone contraction, which could be measured on the sanity-check e
  • My inference: the practical loss trains $Q(s, a+\eta)$ for noise-perturbed in-sample actions, while the target value $\hat{Q}_{\theta'}(s', \pi_\phi(s'))$ for the policy's own actions is left untouched (Appendix D admits this); the actor's OOD evaluations are therefore corrected only indirectly, and a direct version that smooths policy actions falling inside the CHN is a natural extension.
  • My inference: the paper's own noise sweep shows performance collapsing when perturbed actions leave the CHN, so the algorithm inherits sensitivity to the effective neighborhood radius; choosing the noise scale per-dataset from the action-space diameter, instead of a fixed clip, could be a cheap robustness improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper addresses over-conservatism in offline RL by proposing to generalize the Q-function to OOD actions lying in the convex hull of the dataset and a small neighborhood around it (CHN). The authors define the Smooth Bellman Operator (SBO), which replaces OOD backup targets with the Q-value of a neighboring in-sample action, and provide NTK-based safety bounds, an error bound on the empirical Bellman operator, and convergence claims. The practical algorithm SQOG adds a noise-based OOD smoothing term to the TD3+BC critic loss, keeping the TD3+BC actor objective. Experiments on D4RL Gym-Mujoco, Maze2d, and Adroit report state-of-the-art average scores with low runtime, plus ablations on the smoothing weight, the behavior-cloning weight, noise scale, noise clip, and noise type.

Significance. If the theoretical claims held, the paper would make a useful conceptual contribution: it identifies a dataset-geometric region where optimistic OOD generalization is argued safe, and it offers a computationally cheap alternative to generative-model-based OOD handling such as MCQ. The empirical evaluation is a genuine strength: D4RL results are averaged over 4 seeds (and over 8 in Appendix B.6), code is released, ablations cover the main hyperparameters, and the runtime comparison is transparent. The main weaknesses are in the theory-to-algorithm chain and in several proof steps, so the significance is contingent on repairing those gaps.

major comments (5)
  1. [§3.2, Proposition 3 and Eq. (10)] Proposition 3 applies the uniform continuity of the parametric Q_θ (Proposition 2) to the true value function Q^π. The manuscript never proves or assumes a modulus of continuity for Q^π on CHN; in tasks with sharp or step-like Q^π, a nearest dataset action can lie on the other side of a peak, so the premise ||Q^π(s,a_ood)-Q^π(s,a_neighbor)||<ε/2 is not supported. Since Theorem 3 relies on this premise, the claim that SBO drives Q_k toward Q^π is not established for such environments. Please add an explicit regularity assumption on Q^π and empirical evidence, such as Lipschitz estimates from Monte Carlo Q-values, that the assumption is plausible; the sanity check in §4 only evaluates two smooth MuJoCo states and never measures the Lipschitz constant of Q^π.
  2. [Appendix A, Proposition 4] The contraction proof is invalid for OOD state-action pairs. For (s,a) with µ(a|s)=0, Definition 2 gives \hat B^π_2 Q(s,a)=Q(s,a), hence \tilde B^π Q(s,a)=Q(s,a_neighbor) with no discount factor. The proof's OOD branch therefore yields ||\tilde B^π Q_1 - \tilde B^π Q_2||_∞ ≤ ||Q_1 - Q_2||_∞, not γ||Q_1 - Q_2||_∞. The derivation in Eqs. (28)-(30) retains γ on the OOD branch by moving G_1 inside the expectation, which is not justified because G_1 replaces the evaluation point. The operator is nonexpansive in that branch, and convergence to a unique fixed point is not proven as stated.
  3. [§3.3, Eq. (14), and Appendix D] The loss actually optimized by SQOG is not the SBO of Definition 2. In SBO, the OOD target is the neighboring in-sample Q-value; in Eq. (14), the target for the actor-generated action, r+γ min_i \hat Q_{θ_i'}(s',π_φ(s')), is left unchanged, and only a regularizer on Q_θ(s,a+η) is added. Moreover, a+η is not tested for membership in CHN, and the implementation never detects OOD actions. Thus Theorems 2 and 3 do not directly justify the empirical gains. The authors acknowledge this deviation in Appendix D, but the main text should qualify the theory-to-algorithm claim and either modify SQOG to implement SBO or analyze the heuristic loss separately.
  4. [Appendix A, Assumption 1, and §3.1, Proposition 1] All safety bounds (Propositions 1, Theorem 1, and Theorem 2) are proved in the infinite-width NTK regime with infinitesimal learning rate, while SQOG uses finite ReLU networks trained with Adam. Consequently the quantitative safety guarantees do not apply to the deployed algorithm. Please state this limitation in the main text and provide finite-width evidence for transfer, for example width sweeps or a direct check that point-to-dataset distance controls observed Q-differences in the trained networks.
  5. [§3.2, Theorem 3] Theorem 3 assumes Q_k(s,a)≈Q^π(s,a) for all in-sample (s,a)∈D, and Proposition 3 assumes uniform ε/2 accuracy of in-sample Q-values, but no self-consistency or convergence argument establishes this premise. The theorem is therefore conditional on the very accuracy that the algorithm is intended to produce; please reframe it as a conditional contraction result relative to an explicit approximation-error assumption, and state what mechanism in SQOG is expected to maintain that assumption.
minor comments (6)
  1. [§5, Related Work] The proposed algorithm is referred to as 'MQOG' twice in the Related Work section; these should read 'SQOG'.
  2. [§3.2, Definition 2] 'wildly used empirical Bellman operator' should be 'widely used empirical Bellman operator'.
  3. [§2 and Definition 1] The symbol r is used both for the reward bound and for the CHN neighborhood radius; please disambiguate these uses.
  4. [§4, Table 1] The statement that SQOG 'consistently attains the highest scores on most datasets' should be qualified, since SQOG is not the best on hopper-random, walker2d-medium, halfcheetah-medium-replay, and walker2d-medium-expert.
  5. [Appendix B.1] The phrase 'non-Markovian policies' is unclear; please clarify whether the data are generated by time-varying, goal-conditioned, or otherwise non-stationary behavior.
  6. [§4, Figure 1 and Table 1] Please add a colorbar to Figure 1 so the density scale is interpretable, and specify in the runtime comparison whether all methods were run on the same hardware and implementation framework, since Appendix B.7 reports hardware only.

Circularity Check

2 steps flagged · score 5.0 of 10

Partial circularity: the OOD approximation guarantee of SBO assumes the neighbor accuracy it is meant to establish, and the sanity check largely re-encodes the OOD smoothing loss.

  1. other [Section 3.2, Proposition 3 and Theorem 3; Appendix A proof of Theorem 3]
    "Proposition 3 ... For any OOD actions aood within CHN, by Proposition 2, there exist a small δ, if ∥aood − ain_neighbor∥ < δ, then ∥Qπ(s, aood) − Qπ(s, ain_neighbor)∥ < ε/2 ... Theorem 3: assuming that ∥a − ain_neighbor∥ ≤δ and ∥Qπ(s, a) − Qk(s, ain_neighbor)∥ < ε ... ∥Qπ(s, a) − Qk+1(s, a)∥ < ∥Qπ(s, a) − Qk(s, a)∥"

    The SBO defines the OOD update target to be the neighboring in-sample Q-value (Eq. 7), so the substantive claim is that this neighbor value is already close to the true OOD Q-value. Proposition 2 only proves uniform continuity of the parametric Qθ on CHN, not of the true Bellman value Qπ; Proposition 3 substitutes Qπ into that statement without proof. Theorem 3 then assumes exactly the needed proximity (∥Qπ(s,a)−Qk(s,ain_neighbor)∥<ε) and derives the monotone decrease by gradient descent toward a point already assumed ε-close. Thus the central guarantee reduces to the premise that defines the method, rather than being derived from CHN safety or NTK smoothness of Qπ.

  2. fitted input called prediction [Section 3.3 Eq. (13); Section 4, Figure 1 sanity check]
    "LOG(θ) = Es∼D,aood[(Qθ(s, aood) − Q(s, ain_neighbor))^2] ... SQOG consistently achieves accurate estimation of Q-values in most cases."

    The sanity check is presented as evidence that SQOG 'alleviates the over-constraint issue' and accurately estimates OOD Q-values, but the OOD loss literally trains Qθ(s,aood) to equal the in-sample neighbor Q(s,ain_neighbor). On the smooth MuJoCo states used in Figure 1, success of this check is mostly the loss plus environment smoothness, not an independent test of OOD generalization. The D4RL benchmark results (Table 1) are independent evidence and prevent full circularity, but the Q-estimation sanity check itself re-encodes the training target.

full rationale

Most of the paper is self-contained and the benchmark comparisons (D4RL Mujoco, Maze2d, Adroit) are external and not fitted, so the headline empirical claim is not circular. The theoretical chain, however, has a load-bearing step that is effectively assumed: the appropriateness of using a neighboring in-sample Q-value as the OOD target. Proposition 3 claims this follows from Proposition 2, but Proposition 2 is stated and proved for the parametric Qθ (or an abstract continuous Q on CHN), not for the true value function Qπ of the MDP. The true Qπ inherits regularity only from reward and transition structure, which is never established; hence the premise ∥Qπ(s,a)−Qk(s,ain_neighbor)∥<ε in Theorem 3 is exactly the property the paper needs to prove. The proof of Theorem 3 then shows only that moving Qk(s,a) toward a point already ε-close to Qπ reduces error, which is an algebraic consequence of the assumption. The Figure 1 sanity check is also partly built into the loss, since Eq. (13) directly regresses OOD Q-values onto neighbor values. No self-citation from the authors' prior work is load-bearing; the NTK lemmas are cited from DOGE and Bietti-Mairal, and those are external. Overall, score 5 reflects partial circularity in the OOD-approximation argument while acknowledging independent empirical support from the benchmarks.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the NTK idealization, the concentration assumptions, the policy-constraint bound, and the near-accuracy of in-sample Q-values. The algorithm adds tuned constants β, α, noise scale, and noise clip. The CHN is a conceptual set with no computational instantiation.

free parameters (4)
  • β (OOD smoothing weight) = 0.5 for Mujoco/Maze2d; 2.5 for Adroit and halfcheetah-medium-replay
    Controls the weight of the OOD generalization loss in critic update. Tuned per task group based on validation performance (Appendix B.8).
  • α (behavior cloning weight) = 150 for most Mujoco; 25 for Adroit and halfcheetah-medium-replay
    Weights the behavior cloning term in the actor loss. Tuned per task group (Appendix B.8).
  • noise scale = 0.6
    Standard deviation of Gaussian noise added to actions to generate OOD samples; chosen from ablation study (Table 4), not predicted.
  • noise clip = 0.5
    Clipping range for the added noise; chosen from ablation study (Table 4).
assumptions (5)
  • domain assumption NTK regime: two-layer ReLU networks with infinite width trained with infinitesimal learning rate (Assumption 1)
    Used in Lemma 2 and Proposition 1 to bound Q-value differences by input distance; does not hold for finite-width networks with Adam used in SQOG.
  • standard math Concentration properties of reward and transition (Assumption 2)
    Cites Kumar et al. and Auer et al. style assumptions that empirical estimates concentrate; standard in RL theory.
  • domain assumption Policy constraint: max(KL(π,μ), KL(μ,π)) ≤ ε
    Theorem 1 requires the learned policy to be close to the behavior policy; TD3+BC enforces this only weakly via a behavior cloning term.
  • ad hoc to paper In-sample Q-values are near-accurate: ||Q_k(s,a) - Q^π(s,a)|| < ε/2 for all (s,a) in D
    Assumed in Proposition 3 and Theorem 3; the paper claims this follows from prior work, but it is not proven in this paper and is a strong condition.
  • domain assumption OOD actions generated by π lie within CHN
    Used in Proposition 4 convergence proof; in practice the actor loss includes BC, which makes this plausible but not guaranteed.
invented entities (1)
  • Convex Hull and its Neighborhood (CHN)
    purpose: Defines a boundary for safe Q-function generalization in OOD regions
    The CHN is a theoretical construct used for analysis and safety guarantees. The practical algorithm never computes the CHN; instead it generates OOD samples by adding noise, so there is no external falsifiable handle for the CHN itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood." pith.science (2026). https://pith.science/paper/R26Q7EGU

@misc{pith2026250608417,
  author       = {Pith},
  title        = {Pith review of: Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R26Q7EGU}},
  note         = {Machine review of arXiv:2506.08417}
}
abstract

Offline Reinforcement Learning (RL) struggles with distributional shifts, leading to the $Q$-value overestimation for out-of-distribution (OOD) actions. Existing methods address this issue by imposing constraints; however, they often become overly conservative when evaluating OOD regions, which constrains the $Q$-function generalization. This over-constraint issue results in poor $Q$-value estimation and hinders policy improvement. In this paper, we introduce a novel approach to achieve better $Q$-value estimation by enhancing $Q$-function generalization in OOD regions within Convex Hull and its Neighborhood (CHN). Under the safety generalization guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which updates OOD $Q$-values by smoothing them with neighboring in-sample $Q$-values. We theoretically show that SBO approximates true $Q$-values for both in-sample and OOD actions within the CHN. Our practical algorithm, Smooth Q-function OOD Generalization (SQOG), empirically alleviates the over-constraint issue, achieving near-accurate $Q$-value estimation. On the D4RL benchmarks, SQOG outperforms existing state-of-the-art methods in both performance and computational efficiency.

Figures

Figures reproduced from arXiv: 2506.08417 by the authors.

Figure 1
Figure 1. Q-values estimation for two key states. The color bars show the density of different actions. Higher density actions correspond to darker colors. With the tight constraints of the behavior policy, the Q-values of TD3+BC are overly constrained within [-0.50, 0.50] as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. In contrast to MCQ and CQL, our approach significantly reduces computational costs by [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The key features of SQOG and MCQ (Lyu et al., 2022). The use of CVAE makes MCQ time consuming. In contrast, SQOG avoids the use of any generative model, achieving SOTA results with low computational cost. Ablation study. We conduct ablation studies on hyperparameter β and the noise type. The hyper￾parameter β controls the significance of the OOD generalization term in Q-learning, specifically balancing the learning … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Hyperparameter study and noise study on hopper-medium-v2, hopper-medium-replay-v2, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Q-value difference between critic Q-value and true Q-value on hopper-medium-v2, hopper￾medium-replay-v2, halfcheetah-medium-v2. The experiments are run for 1M gradient steps over 4 random seeds. training iterations, indicating that SQOG achieves accurate Q-value estima…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 33 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Uncertainty-based offline reinforcement learning with diversified q-ensemble

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34: 0 7436--7447, 2021

    2021

  3. [3]

    Near-optimal regret bounds for reinforcement learning

    Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008. URL https://proceedings.neurips.cc/paper_files/paper/2008/file/e4a6222cdb5b34375400904f03d8e6a5-Paper.pdf

    work page 2008

  4. [4]

    Manifold topology divergence: a framework for comparing data manifolds

    Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold topology divergence: a framework for comparing data manifolds. Advances in neural information processing systems, 34: 0 7294--7305, 2021

    work page 2021

  5. [5]

    Laplacian eigenmaps and spectral techniques for embedding and clustering

    Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in neural information processing systems, 14, 2001

    2001

  6. [6]

    On the inductive bias of neural tangent kernels, 2019

    Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels, 2019. URL https://arxiv.org/abs/1905.12173

    arXiv 2019

  7. [7]

    Flows for simultaneous manifold learning and density estimation

    Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. Advances in Neural Information Processing Systems, 33: 0 442--453, 2020

    work page 2020

  8. [8]

    Openai gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016. URL http://arxiv.org/abs/1606.01540

    arXiv 2016

Show all 58 references
  1. [9]

    Bail: Best-action imitation learning for batch deep reinforcement learning

    Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 18353--18363, 2020

    2020

  2. [10]

    Diffusion maps

    Ronald R Coifman and St \'e phane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21 0 (1): 0 5--30, 2006

    2006

  3. [11]

    Pink noise is all you need: Colored noise exploration in deep reinforcement learning

    Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=hQ9V5QN27eS

    2023

  4. [12]

    D4rl: Datasets for deep data-driven reinforcement learning, 2021

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021

    2021

  5. [13]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang (Shane) Gu. A minimalist approach to offline reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 20132--20145. Curran Associ...

    2021

  6. [14]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp.\ 2052--2062, 2019

    2019

  7. [15]

    Learning rankings via convex hull separation

    Glenn Fung, Romer Rosales, and Balaji Krishnapuram. Learning rankings via convex hull separation. Advances in Neural Information Processing Systems, 18, 2005

    2005

  8. [16]

    Extreme q-learning: Maxent rl without entropy

    Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328, 2023

    2023 arXiv

  9. [17]

    Improving offline rl by blending heuristics, 2024

    Sinong Geng, Aldo Pacchiano, Andrey Kolobov, and Ching-An Cheng. Improving offline rl by blending heuristics, 2024. URL https://arxiv.org/abs/2306.00321

    2024 arXiv

  10. [18]

    Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters

    Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35: 0 18267--18281, 2022

    2022

  11. [19]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  12. [20]

    Random projections for manifold learning

    Chinmay Hegde, Michael Wakin, and Richard Baraniuk. Random projections for manifold learning. Advances in neural information processing systems, 20, 2007

    2007

  13. [21]

    Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets

    Zhang-Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing...

    2023

  14. [22]

    Mild policy evaluation for offline actor--critic

    Longyang Huang, Botao Dong, Jinhui Lu, and Weidong Zhang. Mild policy evaluation for offline actor--critic. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  15. [23]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl \' e ment Hongler. Neural tangent kernel: Convergence and generalization in neural networks. CoRR, abs/1806.07572, 2018. URL http://arxiv.org/abs/1806.07572

    2018 arXiv

  16. [24]

    A convex hull-based data selection method for data driven models

    Hamid Reza Khosravani, AE Ruano, and Pedro M Ferreira. A convex hull-based data selection method for data driven models. Applied Soft Computing, 47: 0 515--533, 2016

    2016

  17. [25]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.or...

    2015 arXiv

  18. [26]

    Offline reinforcement learning with fisher divergence critic regularization

    Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp.\ 5774--5783. PMLR, 2021 a

    2021

  19. [27]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021 b

    2021 arXiv

  20. [28]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processi...

    2019

  21. [29]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

    2020

  22. [30]

    Kernel metric learning for in-sample off-policy evaluation of deterministic rl policies, 2024

    Haanvid Lee, Tri Wahyu Guntara, Jongmin Lee, Yung-Kyun Noh, and Kee-Eung Kim. Kernel metric learning for in-sample off-policy evaluation of deterministic rl policies, 2024. URL https://arxiv.org/abs/2405.18792

    2024 arXiv

  23. [31]

    When data geometry meets deep function: Generalizing offline reinforcement learning

    Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. arXiv preprint arXiv:2205.11027, 2022

    2022 arXiv

  24. [32]

    Mildly conservative q-learning for offline reinforcement learning

    Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 1711--1724. Curr...

    2022

  25. [33]

    Seabo: A simple search-based method for offline imitation learning, 2024

    Jiafei Lyu, Xiaoteng Ma, Le Wan, Runze Liu, Xiu Li, and Zongqing Lu. Seabo: A simple search-based method for offline imitation learning, 2024. URL https://arxiv.org/abs/2402.03807

    2024 arXiv

  26. [34]

    On the role of general function approximation in offline reinforcement learning

    Chenjie Mao, Qiaosheng Zhang, Zhen Wang, and Xuelong Li. On the role of general function approximation in offline reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=JSS9rKHySk

    2024

  27. [35]

    Machine learning algorithm based on convex hull analysis

    AP Nemirko and JH Dul \'a . Machine learning algorithm based on convex hull analysis. Procedia Computer Science, 186: 0 381--386, 2021

    2021

  28. [36]

    Why is posterior sampling better than optimism for reinforcement learning?, 2017

    Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning?, 2017. URL https://arxiv.org/abs/1607.00215

    2017 arXiv

  29. [37]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  30. [38]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. CoRR, abs/1709.10087, 2017. URL http://arxiv.org/abs/1709.10087

    2017 arXiv

  31. [39]

    Policy regularization with dataset constraint for offline reinforcement learning

    Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, 2023

    2023

  32. [40]

    Nonlinear dimensionality reduction by locally linear embedding

    Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290 0 (5500): 0 2323--2326, 2000

    2000

  33. [41]

    A dataset perspective on offline reinforcement learning

    Kajetan Schweighofer, Marius-constantin Dinu, Andreas Radler, Markus Hofmarcher, Vihang Prakash Patil, Angela Bitto-Nemling, Hamid Eghbal-zadeh, and Sepp Hochreiter. A dataset perspective on offline reinforcement learning. In Conference on Lifelong Learning Agents, pp.\ 470--5...

    2022

  34. [42]

    Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples, 2023

    Hao Sun, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples, 2023. URL https://arxiv.org/abs/2310.07747

    2023 arXiv

  35. [43]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249

    2018

  36. [44]

    A global geometric framework for nonlinear dimensionality reduction

    Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290 0 (5500): 0 2319--2323, 2000

    2000

  37. [45]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033, 2012. doi:10.1109/IROS.2012.6386109

    2012

  38. [46]

    Adaptive manifold learning

    Jing Wang, Zhenyue Zhang, and Hongyuan Zha. Adaptive manifold learning. Advances in neural information processing systems, 17, 2004

    2004

  39. [47]

    Improving generalization in reinforcement learning with mixture regularization

    Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. Advances in Neural Information Processing Systems, 33: 0 7968--7978, 2020

    2020

  40. [48]

    Exponentially weighted imitation learning for batched historical data

    Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018

    2018

  41. [49]

    Behavior regularized offline reinforcement learning

    Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. CoRR, abs/1911.11361, 2019. URL http://arxiv.org/abs/1911.11361

    1911 arXiv

  42. [50]

    Zeta hull pursuits: Learning nonconvex data hulls

    Yuanjun Xiong, Wei Liu, Deli Zhao, and Xiaoou Tang. Zeta hull pursuits: Learning nonconvex data hulls. Advances in Neural Information Processing Systems, 27, 2014

    2014

  43. [51]

    Uncertainty svm active learning algorithm based on convex hull and sample distance

    Hailong Xu, Longyue Li, Pengsong Guo, and Changan Shang. Uncertainty svm active learning algorithm based on convex hull and sample distance. In 2021 33rd Chinese Control and Decision Conference (CCDC), pp.\ 6815--6822. IEEE, 2021

    2021

  44. [52]

    Offline rl with no ood actions: In-sample learning via implicit value regularization

    Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810, 2023

    2023 arXiv

  45. [53]

    Rorl: Robust offline reinforcement learning via conservative smoothing

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35: 0 23851--23866, 2022

    2022

  46. [54]

    Towards robust offline reinforcement learning under diverse data corruption, 2024

    Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption, 2024. URL https://arxiv.org/abs/2310.12955

    2024 arXiv

  47. [55]

    In-sample actor critic for offline reinforcement learning

    Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=dfDv0WU853R

    2023

  48. [56]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  49. [57]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  50. [58]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.