An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning

Amy Zhang; Haoran Xu; Harshit Sikchi; Scott Niekum; Shuozhe Li

REVIEW 4 major objections 5 minor 2 cited by

An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that offline RL is best solved as imitation learning with a corrected, iteratively refined visitation ratio, and that this single mechanism suffices to beat the strongest primal- and dual-RL baselines on every dataset…

desk verdict IDRL is a genuinely useful offline RL algorithm with a clean correction for semi-gradient Dual-RL, but its monotonic-improvement theorem doesn't apply to the implemented filtering step. read the letter →

arxiv 2504.13368 v1 pith:H26CUI4I submitted 2025-04-17 cs.LG cs.AI

Haoran Xu , Shuozhe Li , Harshit Sikchi , Scott Niekum , Amy Zhang This is my paper · ORCID

classification cs.LGcs.AI

keywords offlinereinforcementlearningdualvisitationdistributionratiodiscriminator-weightedbehaviorcloningsemi-gradientfixedpointiterativedatasetfilteringD4RLbenchmarkcorrupteddemonstrations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that offline reinforcement learning — learning a policy from a fixed dataset without further interaction — is best understood as a form of imitation learning: assign every logged transition a weight equal to how much more likely the optimal policy is to take that action than the data-collecting policy was, then clone the data under those weights. It shows that dual RL, the family of methods that tries to learn exactly this visitation ratio, has been estimating the wrong object: a semi-gradient update silently changes the fixed point so that what is learned is an action ratio, which assigns positive weight at states the optimal policy would never visit. The proposed method, IDRL, corrects the ratio in two stages and then exploits the fact that the optimal ratio is sparse — transitions with zero weight are suboptimal and can be removed — by iteratively filtering the dataset and repeating the estimation on the cleaned set. The paper proves that the filtered datasets improve monotonically in attainable return and reports that IDRL matches or beats the strongest primal- and dual-RL baselines on D4RL benchmarks and on corrupted demonstrations where expert transitions are as rare as one percent of the data. If the central claim is right, the practical payoff is a parameter-light offline RL algorithm that needs no expert data, no generative policy model, and no environment interaction, and that gets more stable as it runs more iterations.

What carries the argument

The load-bearing object is the corrected state-action visitation distribution ratio $w^*(s,a) = w^*(s) \cdot w^*(a|s)$ of Eq. 12. It is assembled in two stages: stage one runs semi-gradient Dual-RL (Eqs. 3-4) to obtain the action ratio $w^*(a|s) = \pi^*(a|s)/\mu(a|s)$; stage two feeds that ratio into a Fenchel–Rockafellar dual of a Bellman-flow-constrained off-policy evaluation problem (Eqs. 7-9), yielding convex, unbiased objectives (Eqs. 10-11) for the state weight $W^*(s)$ and the value function $U^*(s)$, with $w^*(s) = \max(0, (f')^{-1}(\mathbb{E}_{a\sim\mu}[w^*(a|s)(T U^*(s,a) - U^*(s))]))$. The second mechanism is the iterative filter: after each pass the dataset is replaced by its positive-weight support, $D_{k+1} = \{(s,a,r,s') \in D_k : w_k(s,a) > 0\}$, which the paper interprets as replacing the behavior visitation distribution with the optimized one; Theorem 4 guarantees monotone improvement in the filtered datasets' attainable return, and Theorem 3 turns that into a behavior-cloning performance bound by viewing iteration $k$ as weighted-BC on an expert set $D_{k+1}$ with a supplementary set $D_k \setminus D_{k+1}$.

What would settle it

Construct a synthetic MDP with a known behavior policy $\mu$ and a known expert policy $\pi_E$, so the true ratios $\pi_E(a|s)/\mu(a|s)$ and $d_E(s,a)/d_D(s,a)$ can be computed exactly. Run IDRL's stage one and its two-stage correction on a dataset drawn from $\mu$, and compare the recovered weights with the true ratios at every state-action pair. If the recovered action ratio is not pointwise close to the true ratio, or if the positive-weight support excludes state-action pairs the true optimal policy visits, then the fixed-point correction and the filtering guarantee that rest on it are empirically false. The same test, applied to the corrupted-demonstration setting, would check whether IDRL's filtered support matches the support of the hidden expert transitions rather than a superset of them.

Watch

Extended reading notes

Core claim

IDRL's central claim is that the optimal discriminator weight — the ratio of the optimal policy's visitation distribution to the behavior policy's — can be recovered from a static dataset alone, without expert data, provided two errors in current Dual-RL methods are corrected. Proposition 1 establishes that semi-gradient Dual-RL converges to the action distribution ratio $w^*(a|s) = \pi^*(a|s)/\mu(a|s)$ rather than the state-action visitation ratio $w^*(s,a) = d^*(s,a)/d_D(s,a)$; the action-only ratio cannot tell whether a state is worth visiting and, under function approximation, generalizes badly to suboptimal states. The correction treats the action ratio as known and recovers the missing state factor by solving a Bellman-flow-constrained off-policy evaluation problem (Theorem 1), with Lemma 1 and Theorem 2 providing unbiased sample-based objectives (Eqs. 10-11); the product $w^*(s,a) = w^*(s) \cdot w^*(a|s)$ is the corrected state-action ratio. Because this corrected ratio is sparse, the regularized optimal policy's support defines a cleaned dataset, and IDRL iterates: filter, re-estimate, filter again. Theorem 3 bounds the weighted-BC suboptimality of the resulting curriculum by the usual imitation gap plus a term that shrinks as the filtered dataset approaches the expert distribution, and Theorem 4 guarantees $V(D_{k+1}) \geq V(D_k)$ — each iteration's dataset is no worse than the previous one in attainable return.

Load-bearing premise

The argument collapses if the action-level ratio learned in stage one is not exactly $\pi^*(a|s)/\mu(a|s)$ — if the semi-gradient fixed point is only approximately the implicit policy ratio, or if finitely many gradient steps on Eqs. (10)-(11) leave $W^*(s)$ inaccurate, then the product $w^*(s) \cdot w^*(a|s)$ inherits both errors and the dataset filter can discard the very transitions the optimal policy needs.

Editorial extensions

If this is right

Any Dual-RL method that trains its value head with a semi-gradient update — IQL, SQL, and their relatives — is learning an action ratio, not a visitation ratio; IDRL's two-stage correction can be grafted onto them as a policy-extraction upgrade.
The sparsity result gives a principled transition-level filtering rule (keep the positive-weight support) that is strictly finer than trajectory-level filtering such as X%-BC, which keeps or discards whole trajectories.
The oracle experiment that motivates the paper — expert-augmented discriminator-weighted BC beating classical offline RL — becomes achievable without any expert data, since iteration replaces the missing expert distribution.
Because Theorem 4 guarantees $V(D_{k+1}) \geq V(D_k)$, practitioners can pick the iteration count by watching the filtered dataset stabilize instead of tuning a regularization strength down, which the paper argues has no monotonicity guarantee and risks overestimation.
On corrupted demonstrations, iterative filtering should recover the rare high-performing transitions that reweighting-based methods miss, which is exactly the regime the paper tests at 1%, 5%, and 10% expert ratios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct diagnostic follows from the paper's own decomposition: train an independent behavior model $\hat{\mu}(a|s)$ on the dataset and check whether $w^*(a|s) \cdot \hat{\mu}(a|s)$ sums to one over actions per state; systematic deviation would flag fixed-point drift in the semi-gradient stage before it contaminates the filter — a test the paper does not run.
The hard-thresholding filter ($w>0$) could be softened into a smooth reweighting with a small floor for low-weight transitions; this would likely reduce the small-data generalization problems the authors list as a limitation, at the cost of a slightly weaker sparsity guarantee.
Nothing in the correction is specific to offline RL: the same action-ratio-to-visitation-ratio fix applies to off-policy evaluation, so the two-stage OPE correction may improve DualDICE-style stationary-ratio estimators in settings the paper never touches.
The monotone-improvement theorem suggests a self-tuning stopping rule — iterate until $V(D_{k+1}) - V(D_k)$ falls within noise — which would remove the manually chosen iteration count $M$ from the algorithm.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I'll give you the short version first: this paper has a real algorithmic idea, and the empirical work is worth taking seriously, but the headline theory overstates what is actually proved. The two-stage correction—learn the action ratio, then solve an OPE problem to get the state-action ratio—is a clean fix for a known semi-gradient distortion, and the iterative support filtering is a sensible way to focus imitation on plausible transitions. The D4RL results and the corrupted-demo experiments show consistent gains with low variance across seven seeds, and the ablation shows both components matter. That alone makes it worth reading.

The soft spots are mostly in the claims, not the method. The biggest one: Theorem 4 proves monotonic improvement for an idealized sequence where the reference distribution is replaced by d*_k, but Algorithm 1 filters by w_k > 0. That gives you the original data restricted to a support, not a resampling from d*_k. So the next iteration's behavior distribution is p_k conditioned on the support, and the theorem's V(D_{k+1}) doesn't track the actual filtered dataset. The paper says "approximately" in the text, but the theorem is stated as a property of the algorithm. That is a genuine gap, and the monotonic-improvement claim should be presented as a heuristic.

Also, the abstract says "beats ... on all datasets," but Table 1 shows several tasks where IDRL is not the best (e.g., walker2d-medium-replay, halfcheetah-medium-expert). The text later says "almost all," so the abstract is just overreaching. The "unbiased estimator" language in Lemma 1/Theorem 2 is only correct in population: the sample argmin is not an unbiased estimate of (f')^{-1}(E[g]) because of the nonlinearity. Minor. And Theorem 3's proof writes D_{k+1}-D_k where it should be D_k-D_{k+1}. Likely a typo, but confusing.

No code is shipped, which makes the empirical claims harder to verify. I'd ask for code.

Who is this for? People working on DICE-style offline RL. It gives a practical recipe and a useful negative observation about semi-gradient updates. I'd send it to review—it deserves referee time, but the theory needs revision or honest relabeling as motivation, not proof.

Referee Report

4 major / 5 minor

Summary. The paper proposes Iterative Dual Reinforcement Learning (IDRL), an offline RL method that views the problem through optimal-discriminator-weighted imitation learning. The authors observe that weighted behavior cloning with a discriminator ratio trained on the offline dataset plus an additional expert dataset performs strongly. They argue that semi-gradient Dual-RL methods learn only an action-level distribution ratio rather than the full state-action visitation ratio, and they propose a two-stage correction: first estimate the action ratio by semi-gradient Dual-RL, then solve an OPE-style objective to recover the state ratio. IDRL then iteratively removes transitions assigned zero weight by the learned state-action ratio and reruns Dual-RL on the filtered dataset. The paper provides theoretical statements about the behavior-cloning bound and monotonic improvement across iterations, a grid-world illustration, and experiments on D4RL and corrupted-demonstration datasets.

Significance. If the central claims hold, IDRL would be a meaningful contribution to offline RL: it identifies a real fixed-point issue in semi-gradient Dual-RL, proposes a decomposition that avoids OOD action queries, and introduces a sparse-filtering mechanism for iterative dataset refinement. The paper's strengths include the explicit derivation of the correction objective, the careful distinction between action-level and state-action visitation ratios, the toy experiment that visually demonstrates the difference, and a fairly broad empirical evaluation including corrupted demonstrations. However, the main theoretical guarantee—Theorem 4—is stated for an idealized sequence of distributions that Algorithm 1 does not actually implement, so the claim that IDRL provably breaks the regularization barrier is not currently supported. The empirical results are promising but the abstract overstates them relative to the paper's own Table 1.

major comments (4)

[§3.2, Theorem 4; Algorithm 1] Theorem 4 does not apply to the implemented algorithm. Algorithm 1 (line 12) constructs D_{k+1} by keeping transitions with w_k(s,a)>0, so the next Dual-RL iteration uses the support-restricted behavior distribution p_{k+1}(s,a) ∝ p_k(s,a)·1_{w_k(s,a)>0}. This is not the optimized visitation distribution d*_k whose ratio w_k=d*_k/p_k was learned: within the retained support, the relative frequencies of transitions are unchanged. The proof of Theorem 4 instead considers d*_{k+1}=argmax_d E_d[r]-αD_f(d||d*_k) and defines V(D_k)=E_{d*_k}[r], concluding V(D_{k+1})≥V(D_k). Thus the monotonic-improvement/curriculum guarantee concerns a sequence of distributions the algorithm never realizes. The paper itself hedges by saying the filtering can be 'approximately viewed' as replacing the behavior distribution, but Theorem 4 is then stated and used as an unconditional guarantee. This gap undercuts the central theoretical claim that IDRL provably breaks the regularization barrier. The authors should either prove a monotonicity statement for the actual filtered-data sequence E_{p_{k+1}}[r]≥E_{p_k}[r] or explicitly and consistently reframe Theorem 4 as a statement about an idealized iteration and remove the claim that the implemented procedure is provably improving.
[§3.1, Theorem 2 and Eqs. (10)-(11)] The claim that W*(s) provides an 'unbiased estimate' of (f')^{-1}(E_{a~μ}[w*(a|s)(TU(s,a)-U(s))]) is not accurate as stated. The result in Lemma 1 identifies W*(s) as the minimizer of a convex expected objective; the argmin of a finite-sample empirical version of that objective is generally biased because the argmin does not commute with expectation under the nonlinear map (f')^{-1}. What is unbiased is the gradient of the empirical objective with respect to W. Since Eqs. (10)-(11) are implemented with minibatches, the paper's later claim that the correction stage recovers the true state-action visitation ratio 'in an unbiased way' is not justified by the stated theorems. Please either correct the terminology and provide a finite-sample analysis, or soften the claim to 'unbiased gradient estimates' and state the resulting bias-variance caveat.
[Abstract; Table 1] The abstract's claim that IDRL 'beats strong Primal-RL and Dual-RL baselines in terms of both performance and stability, on all datasets' is contradicted by the paper's own Table 1. In the antmaze-large-diverse row, ReBRAC reports 54.4±25.1 while IDRL reports 54.2±3.8, so IDRL does not beat that baseline on that dataset. The table caption itself says IDRL 'matches or outperforms' baselines 'on almost all tasks.' The abstract and the main-text summary in Section 4.1 should be revised to match the actual results, removing the 'all datasets' overstatement.
[Theorem 3 proof] The proof of Theorem 3 invokes the imitation-learning bound of Li et al. (2024) by treating D_{k+1} as the 'expert' dataset and D_k \ D_{k+1} as the supplementary dataset, but the required premise is not verified. D_{k+1} consists of transitions kept by the learned weight w_k>0, which are not necessarily expert demonstrations, and the ratio w*_{k+1}(s) used in the bound is not the expert/behavior ratio assumed by Li et al. Additionally, the proof text writes the supplementary set as 'D_{k+1}−D_k', which is empty because D_{k+1}⊆D_k, and should be D_k \ D_{k+1}. Please clarify the assumptions under which the cited bound applies and correct the set notation.

minor comments (5)

[Algorithm 1, line 10] Line 10 says to update Uψ1 and Wψ2 by Eqs. (11) and (12), but Eq. (12) is the formula for the ratio w(s,a), not an update objective. The W update should reference Eq. (10) and the U update Eq. (11).
[Abstract] There is a typo in the abstract: 'addtional' should be 'additional'.
[Section 1] In the introduction, 'uncentainty estimation' should be 'uncertainty estimation', and 'faciliate' should be 'facilitate'.
[Appendix C, Table 3] Table 3 lists λ values for Mujoco and Antmaze tasks but not for the Kitchen tasks, despite Kitchen results appearing in Table 1. Please report the λ values used for Kitchen.
[Figure 3 caption] The caption of Figure 3 says 'this table shows the results' although the results are displayed in a figure; this appears to be a copyediting error.

Circularity Check

1 steps flagged · score 4.0 of 10

Theorem 4's monotonic-improvement guarantee is built into the definition of the regularized-optimum distribution d*_{k+1}, not derived for Algorithm 1's support-filtered dataset; the main correction derivation is otherwise self-contained.

self definitional [Algorithm 1, line 12; Section 3.2, Theorem 4 and its proof]
"Get state-action ratio w_k(s,a) by Eq.(12) and D_{k+1} = {(s,a,r,s′)∈D_k|w_k(s,a)>0} ... Note that V (D) = Ed_D(s,a)[r(s,a)] and because d∗_{k+1} is the solution to d∗_{k+1} = arg max_d Ed(s,a)[r(s,a)] − αD_f [d(s,a)∥d∗_k(s,a)], so ... V (D_{k+1}) = Ed∗_{k+1}(s,a)[r(s,a)] ≥ Ed∗_k(s,a)[r(s,a)] = V (D_k)"

The theorem's conclusion is true by construction for the sequence d*_k, not for Algorithm 1's filtered datasets. d*_{k+1} is defined as the maximizer of E_d[r] − αD_f(d∥d*_k), and d*_k is feasible with zero divergence, so E_{d*_{k+1}}[r] ≥ E_{d*_k}[r] follows immediately from that definition plus nonnegativity of divergences; no property of the filtering operation is used. The implemented D_{k+1} is a support-restricted empirical subset of D_k, with sampling distribution proportional to p_k·1_{w_k>0}; it is neither d*_k nor d*_{k+1}. Hence the 'monotonic improvement'/'curriculum' prediction is a definitional identity about an idealized distribution sequence, and the paper's notation substitutes that sequence for the actual dataset.

full rationale

Most of IDRL's derivation chain is self-contained. The correction stage (Eqs. 7-12) starts from an action-level ratio w*(a|s) and solves the Bellman-flow-constrained OPE problem (7) to recover the state-action ratio; that is a compositional estimator, not a fitted parameter renamed as a prediction. The oracle 'optimal discriminator weight' is defined externally via an expert dataset and is not the algorithm's output. Heavy self-citation is present (Xu et al. 2023; Sikchi et al. 2023b; Mao et al. 2024a), but the key cited property, sparsity of the regularized optimal ratio, also follows from Eq. (5) in this paper, so the citations are not load-bearing. The one genuinely circular step is the theoretical monotonic-improvement claim: Theorem 4 proves V(D_{k+1})≥V(D_k) by substituting the regularized-optimum distribution d*_{k+1} for the Algorithm 1 filtered dataset D_{k+1}; because d*_{k+1} is by definition the argmax of E_d[r]−αD_f(d||d*_k), the inequality is a direct consequence of the definition and does not apply to the actual support-filtered dataset. This is partial circularity in the theory, not in the empirical benchmark results, which are externally evaluated.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The main free parameters are the per-dataset λ and the iteration count M. All theoretical results rest on standard duality and on the offline RL domain assumption. Two ad hoc assumptions are imported into the proof: the applicability of the imitation bound from Li et al. (2024) and the claim that regularized objective monotonicity transfers to final policy performance. No new physical or structural entities are introduced.

free parameters (2)

λ = 0.4 to 0.8 depending on dataset (Table 3)
The regularizer weight λ is selected per dataset by the criterion 'select a larger λ if the value of V doesn't diverge'. It controls the trade-off between reward maximization and divergence penalty in the Dual-RL objective, and the paper identifies it as the single free parameter.
Number of iterations M = 2 for D4RL, 3 for corrupted demonstrations
The iteration count is chosen by dataset type; the paper states D4RL only needs one or two iterations and corrupted datasets benefit from three. This is a free design choice.

assumptions (4)

standard math Fenchel-Rockafellar duality converts the constrained optimization over d in Eq. (7) into the tractable dual objectives in Eq. (9) and Theorems 1-2.
Standard convex-duality results are invoked in Section 3.1 and Appendix B; the paper relies on them to claim w*(s) can be recovered by optimizing U and W.
domain assumption The offline dataset D consists of observations independently drawn from a fixed behavior policy μ, and the environment dynamics P are fixed but only accessible through transitions in D.
This is the standard offline RL premise stated in the Preliminaries; it underlies all in-sample learning and the use of dD as the empirical behavior distribution.
ad hoc to paper The performance bound of Li et al. (2024) for imitation learning from expert plus supplementary data applies directly to the iterative filtering in IDRL, treating D_{k+1} as the expert set and the remaining transitions as the supplementary set.
In the proof of Theorem 3 the paper states the supplementary dataset is D_{k+1} - D_k, which is likely a typo for D_k \ D_{k+1}; the entire bound rests on this mapping, which is not proven in the paper.
ad hoc to paper Monotonic improvement of V(D_k)=E_{d*_k}[r] under the regularized objective in Theorem 4 is a meaningful guarantee for the final policy extracted by weighted behavior cloning.
The proof shows only that the regularized optimal distributions have increasing expected reward; no bound links d*_k to J(π) of the policy learned by Eq. (6).

how reviews work

0 comments

Cite this review

Pith. "Pith review of An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning." pith.science (2026). https://pith.science/paper/H26CUI4I

@misc{pith2026250413368,
  author       = {Pith},
  title        = {Pith review of: An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H26CUI4I}},
  note         = {Machine review of arXiv:2504.13368}
}

read the original abstract

We introduce Iterative Dual Reinforcement Learning (IDRL), a new method that takes an optimal discriminator-weighted imitation view of solving RL. Our method is motivated by a simple experiment in which we find training a discriminator using the offline dataset plus an additional expert dataset and then performing discriminator-weighted behavior cloning gives strong results on various types of datasets. That optimal discriminator weight is quite similar to the learned visitation distribution ratio in Dual-RL, however, we find that current Dual-RL methods do not correctly estimate that ratio. In IDRL, we propose a correction method to iteratively approach the optimal visitation distribution ratio in the offline dataset given no addtional expert dataset. During each iteration, IDRL removes zero-weight suboptimal transitions using the learned ratio from the previous iteration and runs Dual-RL on the remaining subdataset. This can be seen as replacing the behavior visitation distribution with the optimized visitation distribution from the previous iteration, which theoretically gives a curriculum of improved visitation distribution ratios that are closer to the optimal discriminator weight. We verify the effectiveness of IDRL on various kinds of offline datasets, including D4RL datasets and more realistic corrupted demonstrations. IDRL beats strong Primal-RL and Dual-RL baselines in terms of both performance and stability, on all datasets.

Figures

Figures reproduced from arXiv: 2504.13368 by the authors.

**Figure 2.** IDRL in a grid-world domain. The initial state (blue) and the goal state (red) define the task, with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Performance of naive BC, Advantage-Weighted (AW), [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Learning curves of IDRL on D4RL Mujoco locomotion datasets. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

**Figure 5.** Figure 5: Learning curves of IDRL on D4RL Antmaze datasets. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Semi-gradient DICE for Offline Constrained Reinforcement Learning
cs.LG 2025-06 conditional novelty 6.0 of 10

Semi-gradient DICE outputs a policy correction instead of a stationary distribution correction, and CORSDICE recovers the latter to enable accurate cost estimation and safe offline constrained RL.
Dichotomous Diffusion Policy Optimization
cs.LG 2025-12 conditional novelty 5.0 of 10

DIPOLE decomposes a KL-regularized RL objective into a pair of sigmoid-weighted diffusion policies whose score combination (CFG-like) yields stable and controllable policy improvement.

Reference graph

Works this paper leans on

26 extracted references · 11 canonical work pages · cited by 2 Pith papers

[2]

17 Published as a conference paper at ICLR 2025 Lemma

work page 2025
[3]

Idql: Implicit q-learning as an actor-critic method with diffusion policies

Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573,

arXiv
[4]

We haveV (Dk+1)≥V (Dk) after thek-th iteration of IDRL. Proof. Assuming the reward function is bounded, i.e, r(s,a )∈ [0,Rmax]. Note that V (D) = EdD(s,a)[r(s,a )] and becaused∗ k+1 is the solution to d∗ k+1 = arg max d Ed(s,a)[r(s,a )]−αDf [d(s,a )∥d∗ k(s,a )], so we have Ed∗ k+1(s,a)[r(s,a )]−αDf [d∗ k+1(s,a )∥d∗ k(s,a )] = max d Ed(s,a)[r(s,a )]−αDf [d...

work page 2023
[5]

When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618,

Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618,

arXiv
[6]

Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08957,

arXiv
[7]

When data geometry meets deep function: Generalizing offline reinforcement learning

Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022a. Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regul...

arXiv
[10]

Smodice: Versatile offline imitation learning via state occupancy matching

Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433, 2022a. Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i’ll go: Offline goal- conditioned reinforcement learning viaf-advantage regression. arXiv prepri...

arXiv
[15]

Score-based generative modeling through stochastic differential equations

13 Published as a conference paper at ICLR 2025 Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456,

arXiv 2025

Show all 26 references

[16]

A review of off-policy evaluation in rein- forcement learning

Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in rein- forcement learning. arXiv preprint arXiv: 2212.06355,

arXiv
[17]

A policy-guided imitation approach for offline reinforcement learning

Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pp. 4085–4098, 2022a. Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discrimin...

arXiv
[18]

State deviation correction for offline reinforcement learning

14 Published as a conference paper at ICLR 2025 Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xiangyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume ...

2025
[19]

Saformer: A conditional sequence modeling approach to offline safe reinforcement learning

Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: A conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv:2301.12203,

arXiv
[20]

Gendice: Generalized offline estimation of stationary values

Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2019a. Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Deep residual reinforcement learning. arXi...

1905 arXiv
[21]

The regularization term aims at imposing visitation distribution constraints (Nachum & Dai, 2020; Lee et al., 2021; Mao et al., 2024a)

15 Published as a conference paper at ICLR 2025 A A N EXTENDED INTRODUCTION OF DUAL-RL M ETHODS A.1 D ERIVATION OF DUAL-RL Dual-RL algorithms consider the following regularized RL problem as a convex programming problems with Bellman-flow constraints and apply Fenchel-Rockfell...

2025
[23]

Given horizon lengthH and the dataset sample size ofD asND, the behavior cloning performance bound of IDRL at iterationk is given by V (π) = V (Dk+1)−O |S|H 2 NDk+1 +NDk−Dk+1/ maxsw∗ k+1(s) . Proof. The proof is highly built on the Theorem 3 in Li et al. (2024), which gives th...

2024
[25]

The number of total transitions of the noisy dataset is 1, 000,

Table 3:λ used in IDRL Dataset λ halfcheetah-medium-v2 0.5 hopper-medium-v2 0.6 walker2d-medium-v2 0.5 halfcheetah-medium-replay-v2 0.6 hopper-medium-replay-v2 0.6 walker2d-medium-replay-v2 0.6 halfcheetah-medium-expert-v2 0.5 hopper-medium-expert-v2 0.5 walker2d-medium-expert...

2025
[26]

Table 4: Noisy dataset of MuJoCo locomotion tasks with different expert ratios. Env Expert ratio Total transitions Expert transitions Random transitions Walker2d 1% 1,000,000 10,000 990,000 5% 1,000,000 50,000 950,000 10% 1,000,000 100,000 900,000 Halfcheetah 1% 1,000,000 10,0...

2025
[1960]

Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update.arXiv preprint arXiv:2402.00348, 2024a

Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update.arXiv preprint arXiv:2402.00348, 2024a. Liyuan Mao, Haoran Xu, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Diffusion-dice: ...

arXiv
[1970]

Score models for offline goal-conditioned reinforcement learning.arXiv preprint arXiv:2311.02013, 2023a

Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Score models for offline goal-conditioned reinforcement learning.arXiv preprint arXiv:2311.02013, 2023a. Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unificatio...

arXiv
[2015]

Reinforcement learning via fenchel-rockafellar duality

Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01866,

2001 arXiv
[2018]

Off-policy deep reinforcement learning without exploration

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proc. of ICML, pp. 2052–2062,
[2020]

Harnessing mixed offline reinforcement learning datasets via trajectory weighting

11 Published as a conference paper at ICLR 2025 Zhang-Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Harnessing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306.13085, 2023a. Zhang-Wei Hong, Aviral Kumar, ...

2025 arXiv
[2021]

Residual algorithms: Reinforcement learning with function approximation

Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. Machine Learning Proceedings 1995, pp. 30–37,

1995
[2022]

Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.09329,

Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.09329,

arXiv
[2023]

Dealing with the unknown: Pessimistic offline reinforcement learning

Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Dealing with the unknown: Pessimistic offline reinforcement learning. In Conference on Robot Learning, pp. 1455–1464. PMLR, 2022b. 12 Published as a conference paper at ICLR 2025 Ziniu Li, Tian Xu, Zeyu Qin, Yang Yu, and...

2025
[2024]

Selfbc: Self behavior cloning for offline reinforcement learning

Shirong Liu, Chenjia Bai, Zixian Guo, Hao Zhang, Gaurav Sharma, and Yang Liu. Selfbc: Self behavior cloning for offline reinforcement learning. arXiv preprint arXiv:2408.02165,

arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [2]

17 Published as a conference paper at ICLR 2025 Lemma

work page 2025

[2] [3]

Idql: Implicit q-learning as an actor-critic method with diffusion policies

Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573,

arXiv

[3] [4]

We haveV (Dk+1)≥V (Dk) after thek-th iteration of IDRL. Proof. Assuming the reward function is bounded, i.e, r(s,a )∈ [0,Rmax]. Note that V (D) = EdD(s,a)[r(s,a )] and becaused∗ k+1 is the solution to d∗ k+1 = arg max d Ed(s,a)[r(s,a )]−αDf [d(s,a )∥d∗ k(s,a )], so we have Ed∗ k+1(s,a)[r(s,a )]−αDf [d∗ k+1(s,a )∥d∗ k(s,a )] = max d Ed(s,a)[r(s,a )]−αDf [d...

work page 2023

[4] [5]

When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618,

Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618,

arXiv

[5] [6]

Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08957,

arXiv

[6] [7]

When data geometry meets deep function: Generalizing offline reinforcement learning

Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022a. Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regul...

arXiv

[7] [10]

Smodice: Versatile offline imitation learning via state occupancy matching

Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433, 2022a. Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i’ll go: Offline goal- conditioned reinforcement learning viaf-advantage regression. arXiv prepri...

arXiv

[8] [15]

Score-based generative modeling through stochastic differential equations

13 Published as a conference paper at ICLR 2025 Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456,

arXiv 2025