Pith. sign in

REVIEW 4 major objections 4 minor 22 references

Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Online RL fine-tuning can be done from a pre-trained policy alone: a short epsilon-greedy pre-sample stage grows a Q-function from scratch that avoids the pessimism of offline critics.

desk verdict A genuine policy-only fine-tuning setting with a simple warm-up mechanism, honestly scoped but with unquantified headline claims and no released code. read the letter →

arxiv 2505.16856 v1 pith:GBR3HFYL submitted 2025-05-22 cs.LG cs.AIcs.RO

Wei Xiao , Jiacheng Liu , Zifeng Zhuang , Runze Suo , Shangke Lyu , Donglin Wang This is my paper
classification cs.LGcs.AIcs.RO
keywords onlinereinforcementlearningfine-tuningpolicy-onlypre-trainedpolicyQ-functioninitializationoffline-to-onlinebehaviorcloningepsilon-greedywarm-upsoftactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that online reinforcement learning fine-tuning does not need an offline-pretrained Q-function, an offline dataset, or even a pre-trained critic: a pre-trained policy alone is enough. The proposed method, PORL, spends a short pre-sample stage running the frozen policy with epsilon-greedy noise while training a Q-function from scratch at a high update-to-data ratio. That quickly produces a critic whose value estimates are more uniform than those of offline-pretrained critics, which persistently underestimate out-of-distribution state-action pairs. If the claim holds, any imitation-learned or offline-RL policy can be fine-tuned online without the extra artifacts that current offline-to-online methods require. The paper reports PORL matching or beating Cal-QL, IQL, CQL, RLPD, WSRL, and JSRL on Antmaze, Adroit, Kitchen, and RLBench tasks.

What carries the argument

The central mechanism is the pre-sample stage: freeze the pre-trained policy, collect a few thousand environment steps with epsilon-greedy action selection at epsilon = 0.1, and update a randomly initialized Q-network from that replay buffer before any policy update. PORL pairs this with a high update-to-data ratio (UTD = 16), layer normalization on the critic, and an ensemble of 10 Q-networks. The stage produces a Q-function initialization whose value estimates are relatively uniform across in-distribution and out-of-distribution state-action pairs, avoiding both the collapse seen with random critics and the exploration-suppressing pessimism seen with offline critics.

What would settle it

Run PORL on a task where the pre-trained policy has near-zero reward and no high-reward states in its support, using the paper's recommended hyperparameters (epsilon = 0.1, pre-sample steps 5k to 20k, UTD = 16). If final performance matches SAC trained from a random policy rather than exceeding it, the claim that the warm-up critic learns anything useful collapses; the paper itself reports this regime on Adroit-Relocate and Kitchen-complete.

Watch

Extended reading notes

Core claim

The central discovery is that the pessimism of offline-pretrained Q-functions, rather than the absence of offline data, is the main obstacle to online fine-tuning. Empirically, online-trained critics, even those extracted early from a low-performing policy, support faster improvement and higher converged scores than offline-trained critics do when used to initialize fine-tuning. The paper argues this happens because online Q-functions estimate values more uniformly, while offline critics keep underestimating any state-action pair outside the offline dataset even after hundreds of thousands of online steps. PORL operationalizes this by learning its own Q-function during a short epsilon-greedy pre-sample stage from the frozen pre-trained policy, then running standard SAC updates; the result is competitive with offline-to-online methods and uniquely supports behavior-cloned policies.

Load-bearing premise

The warm-up phase only pays off if the pre-trained policy already reaches high-reward regions often enough that a short epsilon-greedy interaction burst lets a randomly initialized Q-function learn useful values before any policy update happens.

Editorial extensions

If this is right

  • If PORL is correct, offline-to-online RL no longer requires a pre-trained Q-function or offline dataset, making the method applicable to any pre-trained policy, including behavior-cloned ones.
  • The persistent pessimistic bias documented in offline Q-functions no longer suppresses exploration during online fine-tuning, because the warm-up critic estimates out-of-distribution values more uniformly.
  • A short warm-up stage (20k steps on Antmaze, Kitchen, and RLBench; 5k on Adroit; epsilon = 0.1; UTD = 16) can replace offline pretraining of the critic as a way to stabilize online tuning.
  • PORL should match or exceed the asymptotic performance of offline-to-online baselines such as Cal-QL, IQL, CQL, RLPD, WSRL, and JSRL on the tested benchmarks while requiring fewer pretrained components.
  • Fine-tuning behavior-cloned policies directly becomes possible, including on sparse-reward manipulation tasks where guide-policy methods and vanilla SAC fail, as shown on RLBench-Closebox.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper experiments only with MLP policies, the most direct extension is to diffusion- or transformer-based behavior-cloned policies; if the warm-up critic transfers to those, policy-only fine-tuning could become a general post-training stage for large imitation models.
  • The paper's comparison suggests the bottleneck in offline-to-online RL is Q-function pessimism rather than missing offline data, which predicts that directly debiasing an offline critic might obtain PORL-like gains without any warm-up stage.
  • The epsilon-greedy warm-up could plausibly be replaced by uncertainty-targeted or entropy-maximizing exploration; if those preserve the uniform Q-estimate property in fewer steps, warm-up length could shrink on narrow-coverage tasks.
  • A testable boundary is that warm-up length should scale with task complexity and policy competence; the paper's ablations already hint that deterministic sampling matches epsilon-greedy when pre-training data coverage is diverse.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies online RL fine-tuning of a pre-trained policy when only the policy is available, with no pre-trained Q-function and no offline dataset. It proposes PORL, which first runs a short pre-sample stage in which the pre-trained policy collects transitions under epsilon-greedy exploration while a from-scratch Q-function is trained with a high update-to-data ratio, LayerNorm, and a Q-ensemble; then standard SAC fine-tuning proceeds. Experiments on Antmaze, Adroit, Kitchen, and RLBench compare PORL with offline-to-online methods (CQL, IQL, Cal-QL, WSRL), an online method with offline data (RLPD), and a policy-prior method (JSRL), and ablations study pre-sample steps, sampling strategies, a BC regularization variant, and UTD ratio. The paper also includes a dedicated section, Sec. 12, stating that PORL is not useful when the pre-trained policy is very weak.

Significance. If the empirical claims hold, PORL is a useful minimal recipe for policy-only fine-tuning, extending online RL fine-tuning to imitation-learned policies where no offline critic exists. The paper is honest about its limitations, reports six-seed main runs, includes a nonstandard benchmark (RLBench), and provides ablations for key design choices. The main caveat is that the central mechanism depends on the pre-trained policy being sufficiently competent, a condition the paper does not characterize beyond the benchmarks selected, and the headline comparisons are presented only as learning curves without numeric final-score tables.

major comments (4)
  1. [Sec. 5.1, Figs. 5-7] The headline comparisons are shown only as learning curves; no table of final scores with means and standard deviations is provided. As a result, claims such as 'PORL achieves superior asymptotic performance compared to baseline methods' in Sec. 5.1 cannot be quantitatively checked, and the paper's key results are not reproducible from the reported data. Please add a final-score table for all tasks and methods.
  2. [Sec. 4 / Sec. 5.2 / Table 1] The pre-sample steps T is a domain-specific free parameter (5k for Adroit, 20k for others in Table 1) that was set using the same benchmark suite later used for evaluation. Fig. 8 shows that this choice strongly affects final performance, yet the paper provides no criterion or validation procedure for selecting T on a new task. This makes the 'minimalist' claim of PORL less persuasive and weakens the independence of the evaluation; please provide a selection rule, a sensitivity analysis over more tasks, or an argument that performance is robust over a range of T.
  3. [Sec. 12 / Alg. 1] The paper concedes that for weak pre-trained policies (Adroit-Relocate, Kitchen-complete), PORL is no better than online RL from scratch. This is not a peripheral limitation: the policy-only setting is most valuable precisely when the pre-trained policy is imperfect and improvable, and the success of the epsilon-greedy pre-sample stage depends on the pre-trained policy having enough competence and coverage to yield a useful Q-function. The load-bearing assumption that a short warm-up yields an accurate, non-pessimistic Q is only verified anecdotally on selected benchmarks. Please provide a more precise characterization of when the warm-up works, e.g., a diagnostic based on policy competence or coverage, or experiments that vary policy quality systematically.
  4. [Sec. 5.3, BC regularization equation] The proposed BC-regularized policy loss is written as L = E[β log π(a_t|s_t) + (1-β)(Q - α log π)]. If this is a loss to be minimized, the +β log π term would decrease the likelihood of actions in the replay buffer, which is the opposite of behavior cloning. If the expression is meant to be maximized, the text should say so explicitly. This sign ambiguity is central to the BC variant in Fig. 10 and must be clarified for the method to be reproducible.
minor comments (4)
  1. [Sec. 3.2] The empirical motivation for preferring online-trained over offline-trained Q-functions is based on a single environment (halfcheetah-medium-v2) and a qualitative UMAP visualization; please state whether similar bias patterns were observed in other domains, or add at least one additional case.
  2. [Sec. 5.1] There is a typo in the text near Fig. 7: 'minipulation' should be 'manipulation', and 'RLbench-Closebox' should be 'RLBench-CloseBox' for consistency with the benchmark name.
  3. [Sec. 6.2] The sentence 'While effective, they does not address scenarios...' contains a subject-verb agreement error; it should be 'they do not address'.
  4. [Sec. 14.3, Table 1] The entry 'Offline Steps 1000,000' should be '1,000,000', and the blank entry for RLBench in that row should be explained (e.g., with a dash and a note that RLBench uses BC-pretrained policies with no offline RL steps).

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: PORL's policy-only fine-tuning claim is supported by external benchmark comparisons, not by a derivation that assumes its own conclusion.

full rationale

The paper's central claim is that a from-scratch Q-function, warmed up by epsilon-greedy interactions with a pre-trained policy, can support competitive online fine-tuning without pre-trained critics or offline data. This is an empirical claim evaluated against external D4RL and RLBench benchmarks, not a formal derivation whose conclusion is assumed. Algorithm 1's pre-sample stage (lines 2-6) collects data with the pre-trained policy and updates a randomly initialized Q-function, but the subsequent SAC updates and final scores are measured outcomes, not constructed equivalences. Section 3's comparison of offline-pretrained versus online-trained Q-functions is motivating evidence, not a self-definitional step: the conclusion that online-style Q-functions have more uniform value estimates is based on UMAP visualizations and empirical curves, and the design choice of epsilon-greedy pre-sampling is a heuristic response to that observation rather than an equation that forces the result. Hyperparameters such as pre-sample steps (5k/20k), epsilon=0.1, and UTD=16 are tuned on the same benchmark suite, which weakens the independence of the evaluation but is ordinary tuning, not circular reasoning. Section 12 honestly concedes that PORL offers no advantage over from-scratch online RL when pretrained policies are weak (e.g., Adroit-Relocate, Kitchen-complete); this limits the generality of the method but does not make the argument circular. The self-citations in the manuscript (e.g., Zhuang et al. 2024, Zhang et al. 2024b, Liu et al. 2023, which share authors with this paper) appear in related-work enumerations and are not load-bearing for PORL's design, its experiments, or its conclusions. No specific reduction from output to input, no fitted parameter renamed as a prediction, and no self-citation chain carrying the central claim could be identified, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method is empirical, so it introduces no invented entities. It relies on several domain assumptions and four tuned numeric hyperparameters, listed above. The pre-trained policy itself is an input from prior training, not invented by the paper.

free parameters (4)
  • Pre-sample steps T = 20k (Antmaze, Kitchen, RLBench), 5k (Adroit)
    Chosen per benchmark after ablations (Fig. 8); the paper gives a qualitative rule (more complex tasks need more steps) but no principled derivation.
  • Epsilon-greedy epsilon = 0.1
    Fixed for all tasks; the paper justifies it only as a small positive value to balance exploitation and exploration.
  • Update-to-data ratio (UTD) = 16
    Set to 16 across all tasks; an ablation compares UTD 4 vs 16 and shows a stability and sample-efficiency tradeoff, but the final choice is heuristic.
  • Q-ensemble size = 10
    Adopted from prior RL works to stabilize Q estimation; no ablation varying the ensemble size is reported.
assumptions (3)
  • domain assumption Offline pre-trained Q-functions persistently underestimate out-of-distribution state-action pairs through the online tuning process, and this pessimism suppresses exploration (Sec. 3.3, Fig. 4).
    This empirical premise motivates discarding the pre-trained critic. It is supported only by a halfcheetah case study and qualitative UMAP plots, not by a formal argument or across-task measurement.
  • domain assumption An epsilon-greedy sampling stage with epsilon=0.1 from the pre-trained policy collects a data distribution broad enough to learn a useful, relatively unbiased Q-function initialization before any policy update (Sec. 4, Algorithm 1).
    This is the core mechanism of PORL. Its validity is task-dependent; the paper's own Sec. 12 notes PORL fails when the pre-trained policy is weak, so coverage of high-reward states is not guaranteed.
  • domain assumption The standard MDP and soft actor-critic framework apply, including access to environment rewards and transitions during online fine-tuning (Sec. 2).
    The problem formulation and algorithm assume an infinite-horizon discounted MDP with reward and transition access, which matches standard continuous-control benchmarks but is not proven for all deployment settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only." pith.science (2026). https://pith.science/paper/GBR3HFYL

@misc{pith2026250516856,
  author       = {Pith},
  title        = {Pith review of: Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GBR3HFYL}},
  note         = {Machine review of arXiv:2505.16856}
}
read the original abstract

Improving the performance of pre-trained policies through online reinforcement learning (RL) is a critical yet challenging topic. Existing online RL fine-tuning methods require continued training with offline pretrained Q-functions for stability and performance. However, these offline pretrained Q-functions commonly underestimate state-action pairs beyond the offline dataset due to the conservatism in most offline RL methods, which hinders further exploration when transitioning from the offline to the online setting. Additionally, this requirement limits their applicability in scenarios where only pre-trained policies are available but pre-trained Q-functions are absent, such as in imitation learning (IL) pre-training. To address these challenges, we propose a method for efficient online RL fine-tuning using solely the offline pre-trained policy, eliminating reliance on pre-trained Q-functions. We introduce PORL (Policy-Only Reinforcement Learning Fine-Tuning), which rapidly initializes the Q-function from scratch during the online phase to avoid detrimental pessimism. Our method not only achieves competitive performance with advanced offline-to-online RL algorithms and online RL approaches that leverage data or policies prior, but also pioneers a new path for directly fine-tuning behavior cloning (BC) policies.

Figures

Figures reproduced from arXiv: 2505.16856 by the authors.

Figure 1
Figure 1. Online RL fine-tuning with pre-trained policy only. The framework overview of PORL [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of actor-critic tuning with randomized [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of online fine-tuning with various Q-function initializations. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Q-Value estimation differences across various Q [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparative experiments on online fine-tuning with [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparative experiments on offline-to-online tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Comparative experiments on online fine-tuning with pre-trained policies trained by Behav [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Impact of pre-sample steps, evaluated every 10k steps and UTD=4. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Sampling methods for the pre-sample stage, evaluated every 10k steps and UTD=4. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Adding a behavioral cloning (BC) term to [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Updating critic’s parameters with higher UTD, evaluated every 20k steps. [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: The resulting state distributions of various Q-function initialization and values of state [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Comparison between PORL and WSRL with pre-trained policies initialization only [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Comparison between PORL (UTD=16) and JSRL. [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: PORL does not work when the performance of pretrained policies is poor. [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Comparison of online RL fine-tuning pre-trained policy with or without retaining offline [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Antmaze tasks visualizations (Umaze, Medium, and Large). [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Adroit tasks visualizations (pen, relocate, and door). [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: Kitchen tasks visualizations. • Overall Description: A multi-stage manipulation benchmark controlling a 9-DoF Franka robot to arrange a kitchen environment into a desired configuration. It requires solving 4 subtasks in sequence, testing combinatorial task planning an…
Figure 20
Figure 20. Figure 20: RLBench tasks visualizations. Below are some additional details regarding the RLBench [James et al., 2020] experimental setup: • ReachTarget (grasping the red ball, see top in [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 5 canonical work pages

  1. [1]

    Is con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657,

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657,

    arXiv

  2. [5]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,

    arXiv 2004

  3. [8]

    Offline reinforcement learning with fisher divergence critic regularization

    10 Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774–5783. PMLR, 2021a. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.0616...

    arXiv

  4. [10]

    Umap: Uniform manifold approximation and projection for dimension reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426,

    arXiv

  5. [12]

    Hybrid rl: Using both offline and online data can make rl efficient.arXiv preprint arXiv:2210.06718,

    Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient.arXiv preprint arXiv:2210.06718,

    arXiv

  6. [13]

    Behavior regularized offline reinforcement learning

    Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361,

    arXiv 1911

  7. [15]

    Policy decorator: Model-agnostic online refinement for large policy model

    Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, and Hao Su. Policy decorator: Model-agnostic online refinement for large policy model. arXiv preprint arXiv:2412.13630,

    arXiv

  8. [16]

    Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

    Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang, Yu Liu, and Wanli Ouyang. A perspective of q-value estimation on offline-to-online reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16908–16916, 2024a. Ziqi Zhang, Xiao Xiong, Zifeng Zhuang, Jinxin Liu, and Donglin Wang. Improving offline-...

    work page Pith review arXiv

Show all 22 references
  1. [17]

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn

    URL https://arxiv.org/ abs/2306.06871. Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705,

    arXiv

  2. [18]

    Imitation learning: Progress, taxonomies and challenges

    Boyuan Zheng, Sunny Verma, Jianlong Zhou, Ivor W Tsang, and Fang Chen. Imitation learning: Progress, taxonomies and challenges. IEEE Transactions on Neural Networks and Learning Systems, 2022a. Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In intern...

    arXiv

  3. [19]

    Reinformer: Max-return sequence modeling for offline rl

    Zifeng Zhuang, Dengyun Peng, Jinxin Liu, Ziqi Zhang, and Donglin Wang. Reinformer: Max-return sequence modeling for offline rl. arXiv preprint arXiv:2405.08740,

    arXiv

  4. [20]

    A closely related problem setting is explored in Jump-start reinforcement learning (JSRL) Uchendu et al

    is a curriculum learning-based online RL algorithm that utilizes a pre-trained policy as a prior. A closely related problem setting is explored in Jump-start reinforcement learning (JSRL) Uchendu et al. [2023], which also begins with a pre-trained policy. However, JSRL treats ...

    2023

  5. [21]

    to retaining offline data, the approach of not retaining offline data generally results in improved average normalized scores, accompanied by increasing variance

    Compared /uni00000013 /uni00000018/uni00000013/uni00000013/uni0000004e /uni00000036/uni00000057/uni00000048/uni00000053 /uni00000017/uni00000013 /uni00000018/uni00000013 /uni00000019/uni00000013 /uni0000001a/uni00000013 /uni0000001b/uni00000013 /uni0000001c/uni00000013/uni0000...

    2024

  6. [22]

    In our experiments, the pre-trained policy serves as the policy prior

    is a curriculum learning-based online RL algorithm that utilizes a pre-trained policy as a prior. In our experiments, the pre-trained policy serves as the policy prior. All baseline implementations leverage publicly available codebases with default hyperparameters to ensure re...

    2022

  7. [2009]

    doi: 10.1016/j.robot.2008.10.024

    ISSN 0921-8890. doi: 10.1016/j.robot.2008.10.024. URL https://doi.org/10.1016/j.robot.2008.10.024. Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1...

    2008 doi

  8. [2018]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359,

    2006 arXiv

  9. [2019]

    Elastic decision transformer

    Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. arXiv preprint arXiv:2307.02484,

    arXiv

  10. [2020]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246,

    arXiv

  11. [2021]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learni...

  12. [2022]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,

    2005 arXiv

  13. [2023]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch

    URL https://api.semanticscholar.org/CorpusID:266598468. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural i...

    arXiv

  14. [2024]

    Dongxiang Chen and Ying Wen

    URLhttps://arxiv.org/abs/2410.24164. Dongxiang Chen and Ying Wen. Dcac: Reducing unnecessary conservatism in offline-to-online reinforcement learning. Proceedings of the Fifth International Conference on Distributed Artificial Intelligence,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.